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I. SoMary of Planning ActlTltles and Evaloation Report 

A consortium of four institutions, based at Western Michigan University, 
was established to plan a Research & Development Center on Student Testing, 
Evaluation, and Standards Setting. This consortium is committed to a long 
period of collaboration on research and development addressing important 
is'sues in testing, evaluation, and standard setting. The four consortium 
sites are Boston College, the Dallas Independent School Di^jtrict, the Uni- 
versity of Kansas, and Western Michigan University. 

This report is a final performance report for a planning grant made to 
Western Michigan University by the National Institute of Education. The 
report is required to fulfill conditions of the grant, as described in Grant 
Announcement No. PA-84-3. This first section provides a summary of the plan- 
ning activities actually conducted under the award to Western Michigan Uni- 
versity. It includes a description of particular problems and successes, a 
list of participants and their affiliations, and an evaluation report for the 
planning project. The second section is a technical report on the R and D 
mission for the Center. It updates the mission and strategy statement con- 
tained in the planning grant application and includes an agenda, with support- 
ing justification for R & D within the Center's mission. The third section 
contains a futures paper, as required by the grant announcement. It is in- 
tended for practitioners and researchers and describes work needed in the 
mission area of the Center to accomplish desirable goals by 1990. A biblio- 
graphy of sources used in the planning project, and supporting documentation 
for particular parts of the planning project arc appended to this performance 
report. 

The consortium has an exemplary record in working with state education 
agencies (SEAs) and local education agencies (I,EAs) in furthering educational 
development derived from integrated research. Western Michigan University's 
(WMU) Evaluation Center, under the direction of Drs. Stufflebeam, Sanders, and 
Bunda, has a long and successful history of R & D at SEAs and LEAs and, in 
particular, the development of minority and special client programs. Boston 
College's (B.C.) Center for the Study of Testing, Evaluation, and Educational 
Policy (CSTEEP), under the leadership of Dr. Madaus, became seriously involved 
with policy issues through an extensive process of research at local, nation- 
al, and international levels. Dallas Independent School District's (DISD) 
Department of Research, Evaluation and Information Systems, under the leader- 
ship of Dr. Webster, is home for the premier local educational agency R&D 
center in the world. The University of Kansas' (U of K) Center for Educa- 
tional Testing and Evaluation, directed by Drs. Poggio and Glasnapp, his 
performed a state government mandated assessment and development of student 
testing throughout this heartland state. The consortium has also secured the 
involvement of two top psychometricians (Drs. Hambleton and Swaminathan from 
the University of Massachusetts at Amherst) to add to the scope of its R & i? 
efforts. 

The objectives of the planning grant were to: 

1. Create the goremauce stmctnre, the admlnlstratlTe structare, and 
the fonal agreeaents to undergird the censor tlcn 



2. Explicate the proposed Blsslon acd iwiedlate and long tent objectives 



3. Develop and reach agreeaent on an agenda of projects 



4. Develop staffing facilities and work plans for the initial effort 

5. Act afflxwtlvely in the recrultaent of ainorltles and voaen to key 
staff positions in the Center 

6. Develop a netirork of cooperating ageadea to help plan, conduct, and 
use the Center's projects 

7. Develop a plan and arrangeaents for an ongoing evaluation of the 
Center 



The first goal of the consortium was to secure letters of institu- 
tional support and formal agreement from each member institution. This was 
done concurrently with the development of the formal governance structure 
since the same people, university presidents, had to be contacted. Having 
achieved this commitment, the director. Dr. Stufflebeam, convened the con- 
sortium mambers and school district superintendent in Chicago following the 
annual American Educational Research Association/National Council on Measure- 
ment in Education (AERA/NCME) convention to discuss the specifics of adminis- 
trative structure. It was there that the decision to incorporate a deputy 
director. Dr. Thompson, into the administrative structure was made to 
strengthen th*. Center's day-to-day administration. The identification of an 
executive committee, the individual site directors (Drs. Madaus, Poggio 
Hambleton, Webster, and Sanders), to share in the administrative responsi- 
bilities of the Center helped to create, an atmosphere of collegiality and 
trust. ^ 



The successful operating of the Center was seen to be contingent upon the 
establishment of a capable system of governance and administration. Admini- 
stration would be responsible for the operations and quality of R & D pro- 
ducts. It would coordinate these activities and provide for the temporal and 
fiscal efficacy of effort and product. It would insure that the needs of all 
concerned populations were addressed. Central to the development of admini- 
strative expertise is the requirement of an ongoing evaluation of R & D and 
all if its attendant activities. The evaluation component of this consortium 
wxlX provide information to its national advisory panel to assist it in di- 
recting and redirecting the thrust and scope of its R & D activities. The 
evaluations will, also, be instrumental in determining the success of the 
consortium in meeting its interim and ultimate objectives in disseminating 
results. External evaluations will serve to validate the internal evaluation 
process and further the practice of relevant R&D. Together, the National 
Advisory Panel, the Governance Board, and both internal and external evalua- 
tors will work to insure that en exemplary standard of ethical practice in 
research, development, and dissemination is adhered to. The Governance Board 
will participate in the inevitable political decisions that must occur in any 
cooperative enterprise. Its function will involve the authoritative alloca- 
tion of resources across programs to insure a viable collaborative organiza- 
tion. Consortium site representation will be equal and institutional respons- 
ibility and support enhanced by the B^ard. The governance and administration 
structure created to achieve the first objective of the planning grant is 
provided in Figures 1 and 2. «- e. & 



Figure 1. General Structure of the Center 
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The planning committee, composed of the director, deputy director, execu- 
tive committee, and research panel was then formed to produce and justify the 
Center mission and Its short and long term goals. The research panel con- 
sisted of researchers from the member Institutions, cooperating agencies, and 
Independent agencies. A needs assessment. In the form of a key Informant 
survey, was conducted to Insure the relevance and efficacy of the proposed 
Center mission. Respondents Included officials from such organizations as the 
National Urban League, Council for Basic Education, American Association of 
School Administrators, Mexican American Legal Defense Fund, Southwest Educa- 
tional Development Laboratory, .Public Education Association, Center for 
Teaching & Learning, National Education Association, as well as a number of 
SEA and LEA practitioners. An analysis of responses from this survey was 
paired with a content analysis of the original proposal to best find a match 
betweea what wap needed and what the consortium could best accomplish (See 
Appendices 1 through 6). The result was an extensive array of problem sets. 
From this array was chosen a group of programs that addressed the needs that 
emerged from planning activities. 

The consortium examined needs of the educational community that pertained 
to the Center's proposed mission. This was done through a review and analysis 
of literature, and through Interviews and a survey of key Informants. The 
interviews were exceptionally influential in the conceptualization of needs 
for R & D on student testing, evaluation, and standards setting. The litera- 
ture review was performed by staff personnel best qualified in the areas to be 
addressed. 

The next step in this planning process was the submission of position 
papers dealing with the most critical aspects of the programs. These concept 
papers were offered by consortium staff members and research panel members to 
the planning committee as what each researcher viewed as the most important 
problems and appropriate research and development projects. The papers were 
distributed to each site and discussed via CompuServe and twice ronthly con- 
ference calls. Approximately twenty different concept papers were developed 
in this manner for consideration by the planning committee. The papers went 
through a number of drafts and revisions by the original authors, as well as 
by members of the planning committee. These papers were the basis for creat- 
ing a program agenda. A summary of the concept papers is appended to this 
report. 

A second meeting of key personnel was held at the Evaluation Center at 
WMU in July to make critical decisions for the final proposal. At this 
meeting a great deal of work was done in finalizing the agenda of proposed 
projects and allocation of resources. The ability of the consortium to per- 
form effectively and efficiently was demonstrated at this time. If ever there 
was a time for a breakdown of process to occur it was during this crucial 
stage of negotiation in setting priorities. The effectiveness of the organi- 
zational structure of this collaboration was apparent in the deliberative 
process and in the resulting agenda of projects that was established. Agree- 
ment on the agenda was brought about by orderly processes set in place through 
the establishment of the executive committee. 

There were three distinct selection processes which took place during the 
term of this planning grant; 1) pcograa selection and 2) project selection, 
described above, and 3) personnel selection, to follow. 



The selection of R & D personnel was an organic proces^i of melding a 
diverse group of experts with unique concerns and perspective into a syner- 
gistic community of educational scholars with a broad purview and common 
goals. To achieve this complex organizational structure, an alliance of all 
members had to be established. This alliance had to provide for the indivi- 
dual staffing needs of site projects and at the same time offer the Center 
director the ability to make budget and staffing projections and to maintain 
the cohesiveness of the Center's R&D programs. 

It was initially the responsibility of Dr. Stufflebeam, the proposed 
Center director, to compose a team of researchers able to achieve the goals 
set forth in the RFP. To this end he invited colleagues who haj a history of 
productive R&D experience, and previous collaboration experiences to respond 
to WMU's proposal for a planning grant for the R & D center. Of those re- 
sponding it was determined that the greatest strength of past accomplishment 
and present capability was to be found at Boston College, the Dallas Independ- 
ent School District and the University of KLJsas. Since the inception of the 
proposal process. Dr. Madaus at B.C., Drs. Denton and Webster at DISD, Dr. 
Poggio at U of K, and Dr. Stufflebeam have labored to bring together that team 
of researchers best qualified to achieve the R & D Center goals. Very early, 
it became obvious psychometric theory input would be of fundamental importance 
in the Center's work, and Drs. Hambleton and Swaminathan at the University of 
Massachusetts, Amherst, were added to the team roster. Each member of the 
total team has a history of excellence and delivery of research products. 
Their institutions provided letters and pledges of institutional and financial 
support. This includes the commitment of additional key personnel from each 
member institution. 

The determination of this group to create and sustain an excellent re- 
search organization led them to recruit, not only, the brightest lights at 
their own organizations but the best available talent throughout the country 
to augment and enhance the consortium's effort. They asked Dr. Tyler to serve 
as he^ad of the National Advisory Panel because it was believed that Dr. 
Tyler's unmatched experience in educational R&D related to student testing 
would be invaluable in providing advice to the consortium in area's both 
practical and esoteric. Dr. Tittle was also sought to contribute her particu- 
lar expertise in evaluation and concerns of special populations to the chair 
of the external evaluation panel. The significance of these two appointments 
is reflected in the national prominence these two hold in the field of educa- 
tional R&D related to testing. Their appointments also bolstered the abil- 
ity of the consortium to plan significant and technically sound programs of 
research. 



A great many other notable individuals have concurred with the objectives 
set forth by the consortium and have pledged to assist in planning and imple- 
menting projects. These include Drs. Cuba, Austin, Cooley, Diamond, Iverson, 
Mayo, Scriven, Sandberg, Pullin, Merwin, Wallace, Lincoln, Travers, and Rao, 
who have each contributed substantial time and resources to the project thus 
far. Letters of cooperation, communication, and collaborative intent were 
solicited and received from sixteen recipients of NIE R&D Center planning 
grants, six recipients of NIE Laboratory planning grants, fourteen national 
and international consultants and R&D organizations, and about fifty school 
districts. 



While the Individual plan of action in the planning stage of this propo- 
sal was developed on site, the finished product (the proposal) reflects a 
united effort by all the consortium personnel and a considerable amount of 
support from outside but interested parties. This support at WMU included, 
but was not limited to, Mr. Wil Emert, Division of Research and Sponsored 
Programs; Mr. Frank Jamison, Media Services; Mr. James Kirklin, Academic 
Services Technical Support; Ms. Jane Bauer, Office of Conferences and 
Institutes; and Ms. Janice Argue, Grants and Contracts. 

Affirmative action plans and records are on file at the WMU Evaluation 
Center for Boston College, the Dallas Independent School District, the Uni- 
versity of Kansas , the Universi ty of Massachuse tts , and Wes tern Michigan 
Univo>rsity. Each institution is formally committed to policies of recruitment 
and appointment that provide equal employment opportunity to all qualified 
persons , in all job classif ica tions , in recrui ting , hiring , employment up- 
grading, promotion, selection for training, transfer, termination, compensa- 
tion, benefits, and working conditions, without regard to race, color, reli- 
gion, age, sex, national origin, veteran status, or handicap. The consortium 
members have developed and identified strategies necessary to recruit key 
staff people not namad in the proposal. These staff positions will be filled 
in accordance with the letter and spirit of all civil rights and equal employ- 
ment opportunity legislation, paying particular attention to the hiring and 
professional development of minorities, women, the handicapped, and the 
elderly. The positions to be staffed include key line staff posts, members of 
the National Advisory Panel, consultants, visiting scholars, collaborators in 
research, and support staff. 

The construction of governance and administration policy was of major 
import in the development of a strong R&D organization. While the consor- 
tium recognized the need for this capability it still felt that its investi- 
gatory efforts should be focused on R & D. It quickly became apparent that 
the administrative component was necessary and complementary to the R&D 
mission. What emerged was a strategy for allocating resources primarily for R 
& D, and secondarily for activities involving governance, evaluation, and 
administration. These activities became support mechanisms to the R&D, 
collaboration and dissemination efforts. 

Fiscal and physical resources were assessed at each site in light of 
their ability to contribute to the collaboration/dissemination effort. Papers 
on previous collaborative R&D organizations were studied and discussed. 
Professionals in the field were consulted for their insights and Inputs. 
Practicing educators were requested to add their ideas about R&D strategy. 
From this information emerged a strategy of dissemination involving compre- 
hensive Collaboration, communication, and professional development relating to 
educational research and development. This collaboration includes a network 
of cooperating agencies already taking form as noted above* Those agencies or 
individuals who either have sent letters of collaborative intent or who par- 
ticipated in the key Informant survey have certainly Influenced the scope and 
direction of research and development proposed by this consortium. In this 
manner it may be said that they have already assisted in the planning of the 
Center^s programs and projects and they have in their correspondence and },n 
personal communications indicated their willingness and desire to participate 
in the conduct of research and its subsequent application and evaluation. 



The seven objectives of the planning grant were evaluated according to 
the following criteria: 

1. Manageaent of the project— was it carried out as designed, in a 
timely and cost-efficient manner, with reports and deliverables 
submitted as planned? 

2. Scope of the prolect— were the appropriate individuals and groups 
involved in a meaningful way in the planning of the Center? Were the 
sources of information appropriate for the information that was 
needed? Were any shortcomings in input, due to procedures or sources 
that were employed, rectified? 

3. Use of Infozutlon In planning the CenteT--was information gathered 
during the planning process used in developing the final plan? Can 
the final plan be justified using the available planning information? 

4. Fiscal responsibility— was the grant support used to support import- 
ant planning work that otherwise might not have been feasible? 

The evaluation process for the planning grant involved extensive needs 
assessment (context evaluation), and reviews of drafts designs, concept 
papers, and draft sections of the proposal for an institutional grant (input 
evaluation). The context evaluation included an extensive review of pertinent 
literature which was reported in the proposal for an institutional grant and 
summarized in the second section of this final performance report. A biblio- 
graphy of all sources found to have some bearing on the needs for this Center 
is also appended to this report. The context evaluation also included inter- 
views with educators and researchers about needs that should by addressed by 
the national center. The interviews were reported in the form of memoranda to 
the files which were then used by the planning team to conceptualize proposed 
programs and projects. Finally, the context evaluation included a survey of 
key informants to obtain their perceptions of need and priorities for the 
Center. The results of this survey are provided in Appendix 3 of this final 
report. The key informants were selected because of their knowledge of issues 
faced by traditionally under represented groups and of issues that reflected 
national concerns. The context evaluation resulted in the identification of 
needs that were reviewed by the planning team. These needs became the basis 
for program planning. 

Input evaluation was done at each step of the planning process. Alterna- 
tive designs were solicited froa key personnel and consultants for each compo- 
nent of the later plan. The planning team then served as evaluators who 
submitted their recommendations on designs, program specifications, and drafts 
of the institutional grant proposal to the Center director. In addition, 
content analyses of reviews of the WMU planning grant applications and of 
Grants Announcement No. PA-84-3 were performed to guide design decisions. 
Appendices 5 and 6 of this report reflect the findings of the content 
analyses. 

The director, Dr. Stufflebeam, and his staff at WMU then used these 
evaluations to select or revise each part of the institutional grant proposal. 
Designs often went through several cycles of this evaluation process. Thus, 
the institutional grant proposal has been evaluated systematically by key 
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personnel and consultants. Individuals who were involved in the ulannine 
process are listed in Appendix 1. The planning project was carried out as 
planned with rigorous attention given to identification of needs and to ob- 
taining and getting reviews of a wide range of alternative designs for each 
component of the institutional grant proposal. Drs. Sanders and Stufflebeam 
worked closely to assure that the final proposal for the Center: 

- was based on the planning process that was described in the 
WMU application for a planning grant 

- used input from the important constituencies listed in the 
application for a planning grant 

- was a valid representation of high priority needs in student 
testing, evaluation, and standard setting 

- made efficient and effective use of planning grant support 

The evaluation plan for the proposed Center was also reviewed using the 
" . Evaluations of Educatio n al Programs. Projects and Ma ter ia Is , 

published by the Joint Committee on Standards for Educational Evaluation . 

The operational strategy for the planning project lias closely paralleled 
the strategy that we foresee for the operation of the Center itself. Specifi- 
cally, we have: *^ 



ca 

1. 



Identified critical needs in schools that should be addressed by the 
Cen ter 



2. Involved qualified practitioners, statespersons, and researchers In 
the planning of long-term programs of research and (-velopment to be 
undertaken by the Center 

3. Arranged for school-based research and development settings that 
represent the diversity of settings to be found in the United States 

4. Arranged for collaboration with staff members of local school 
districts, regional laboratories and other research and development 
Centers, professional associations, state education agencies, and 
other relevant educational agencies to accomplish the work of the 
Center 

5. Established communication channels to disseminate the wotk of the 
Cen ter 



II. Report on Research and Developaent Mission 

In compliance with the requirements of Grant Announcement No. PA-84-3, 
this technical report on the research and development mission for a proposed 
NIE Center on Student Testing, Evaluation, and Standard-Setting has I>een 
prepared e.nd is hereby submitted as part of the final report for th«. Western 
Michigan University Planning Grant. This technical report updates the mission 
and strategy statement contained in the planning grant application. It also 
includes an agenda, with supporting justification, for research and develop- 
ment within the Center's mission. This report contains information that was 
in the Western Michigan University institutional grant proposal. The intended 
audience for this report is the National Institute of Education with the 
understanding that NIE may make it available to the recipient of the Center 
institutional grant and to other researchers and practitioners in the mission 
area of the Center. 

The planning period between January and August, 1985 provided an oppor- 
tunity to compile research findings within the Center's mission, tc collect 
information about practitioners' needs, and to reflect on the most appropriate 
allocation of limited research and development resources that would address 
and build upon our findings. 

In this first section of the Technical Report on Research and Development 
Mission, we discuss the rational for the mission statement provided in our 
institutional grant proposal. In the second section, we discuss reasons for 
the selection of the Research and Development approaches that were described 
in our institutional grant proposal. 

Mission for the Center 

We recognized from the outset of the planning project that the funds for 
the proposed center would be limited and that we had to converge on certain 
needs if the gains from research and development are to have practical signif- 
icance. Our reviews of past research and our discussions with practitioners 
led us to focus our plans on the conduct of research and development on 
testing. *^ 

Testing of students has been a part of education for several millennia. 
Prior to the nineteenth century, testing was usually conducted via oral exam 
or some sort of performance examination. In the mid-nineteenth century, 
written exams came to be widely used in the schools of the United States. 
Then in the late nineteenth century and early twentieth century, with the 
founding of psychology and the huge growth in the enterprise of schooling, 
various forms of achievement, psychological, and physiological testing were 
introduced in the schools of America. 

Historically, the main purpose of student testing has been the evaluation 
of student learning. in recent decades, however, testing in the schools has 
taken on additional functions. In the 1930s testing was advocated and in- 
creasingly used as a source of information for student guidance and selection 
into college. In the 1950s new testing programs were introduced as a means of 
identifying exceptional children with special, school needs. In the 1960s and 
1970s testing came to be widely used as a means of program evaluation. In the 
1970s, many new state-wide testing programs were also introduced as a means of 
ensuring the accountability of schools, to help guarantee that students were 



at least minimally competent In the basic ski! Is, and to monitor grade- to- 
grade promotion and high school graduation. Nok ta the 1980s new testing 
programs are widely advocated as instruments of educational reform. Examples 
are interactive computer tests, content-relevant tests, and curriculum- 
embedded tests, which are heavily dependent on advances in psychometric test- 
ing theory . 

In short, student testing in the schools of the United States now serves 
an extremely wide variety of purposes and functions. Among these are: 

- evaluating student learning 

- selecting students for placement into special programs or institutions 

- evaluating educational programs 

- identifying students with special needs 

- improving educational standards 

- helping to guarantee the minimal competency of students 

- guiding instruction 

- informing student guidance and counseling 

- monitoring grade-to-grade promotion and high school graduation 

- helping to ensure the accountability of the schools 

- serving as a tool for educational and psychological research 

- communicating with parents regarding student progress 

there are, of course, many other roles and ways of describing the roles 
that student testing currently serves in the schools of the United States, but 
even this brief listing indicates three general points: 

1. Student testing is serving many different purposes and functions; 

2. The functions and purposes served by testing seem to be increasing 
over time; 

3. Testing is affecting education at many different levels (i.e., through 
teachers, administrators, students, parents, schools, and school 
districts via state and national educational policy; the results of 
testing and the growth of its use are influencing the very ideas and 
constructs we use to ^"^Ink about schools and education). 

Given such diversity, what then should be the mission of the NIE Center 
on Student Testing, Evaluation, and Standards? Our answer to this question is 
premised on six key assumptions, derived from our reading of the history of 
testing and research on testing and from our extensive experience in working 
with schools to use testing and evaluation to improve teaching and learning. 
These assumptions are: 

(1) Educational research on testing is most likely to be productive in 
improving schooling if it is conducted in close collaboration with 
local schools, teachers, and adainlstzators. A tremendous amount of 
experience over the last three decades has shewn that one of the 
biggest problems with all educational research is that of translating 
its findings into educational practice. One clear remedy to this 
problem is to involve schools and educational practitioners in re- 
search from the start: from its inception to its execution to its 
interpretation. Thus, one key assumption underlying our proposal is 
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that research on testing should be conducted collaboratively with 
schools and school people across the nation. We think the consortium 
we have put together will facilitate that collaborative effort. 

Most of the testing over the last 80 years has been based on what Is 
widely called the classical test model and employs paper-and-pencll 
multiple-choice testing. Recent technical developments In both test 
theory and In electronic and video Information processing, however, 
offer much potential to use new techaology to improve many of the 
functions serr-ad by testing. This will allow us to break free from 
both the classical test model and the paper-and-pencll multiple- 
choice format. 



Current models of testing using the paper-and-pencll multiple-choice 
format are so widely used that It surely will be a fairly long time 
(i.e., at least five years) before new testing technology can signif- 
icantly affect ths testing practices of the majority of schools In 
the U.S., but the need for Improving education is too urgent to rely 
exclusively on the long-range strategy of using new technologies for 
improving testing. Another key assumption, therefore, is that the 
work of the Center must also encompass research aimed at making 
currently aTailable test information more useful. 

In planning a research program for an enterprise as large and diverse 
as that of educational testing, some means of differentiating func- 
tions of testing must be employed. A laundry list of different 
functions served by student testing la not a very economical way of 
drawing such a distinction. We think that a useful &na fairly fsrada- 
mental distinction is between external, or policy-oriented testing, 
▼ersus internal, or school-oriented testing programs. The external 
versus internal distinction refers to the locus of initiation for the 
testing. External testing programs are those, for example, mandated 
by national or state educational agencies. Such testing programs may 
also be r-f erred to as policy-oriented because they are providing 
information for educational accountability programs and for policy 
makers that transcends individual schools. In contrast, internal 
testing programs are those undertaken as a matter of discretion by 
individual school systems. Such internal testing programs may use 
externally produced tests but also Include teacher-made tests, and 
systematic observation. 

There are two main reasons for drawing this distinction. First is 
the way in which testing tends to be carried out. External or 
policy-oriented testing programs are carried out on a relatively 
large scale, and use the paper-and-pencll multiple-choice format and 
are machine scored. Though this format of testing may also be used 
in internal testing programs, the latter type of testing also often 
includf.s other methods of assessment, such as fill-in- the-blank , 
short answer, and essay tests. Second is the way in which these two 
types of testing programs affect teaching and learning. Internal 
testing programs may affect teaching and learning quite directly, for 
instance how an individual teacher Instructs an lndlvid»\al student. 
In contrast, external testing programs affect teaching and learning 
indirectly, for example via changes in formal curricula and school 
organization. 
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(5) Testing may be the most visible instrument and indicator of educa- 
tional standards and the most widely recognized means of student 
evaluation, but educational standards are also embodied in many less 
obTioas ancf less tangible forms. Examples of these other less tang- 
ible embodiments of educational standards are curriculum require- 
ments, teachers' and parents' expectations and judgments regarding 
student learning, and even the way time is allocated in schools. 
Given that the ultimate aim of the work of the Center is improved 
student learning and higher educational standards through improved 
testing practices, testing cannot be viewed in isolation from other 
foras in which educational standards are embodied. 

(6) Given the fact that testing serves so many different functions af- 
fecting education on many different levels, and is only one embodi- 
ment of educational standards, the research of the Center must employ 
multiple methods of research and multiple research perspec tires. 

The mission for the Center that was presented in our proposal for an 
institutional grant flowed from our six key assumptions. The mission state- 
ment was as follows: 

This Center will contribute to the improvement of the schools of the 
United States by conducting a collaborative program of research and develop- 
ment directed at improving student testing, evaluation, and standard setting. 
The Center will directly benefit policy makers, educators, parents, and stu- 
dents by (a) enhancing appropriate and fair use of currently available tests 
and (b) developing new and improved evaluative procedures, instruments, and 
systems. The efforts will be focused on needs and problems at both state and 
local levels. 



In light ot the key assumptions that we made, the overall mission of the 
Center will be to conduct research and development on testing: 

- in close collaboration with SEAs and LEAs and intensive involvements 
with selected schools 

- by taking advantage of and promoting the potential of new developments 
in psychome tries and new technologies 

- while at the same time seeking to improve the utility of currently 
available test information 

- with regard to both external and internal testing programs 

- by viewing testing in conjunction with other embodiments of educational 
standards 

- using multiple methods and perspectives of research 

all, with the ultimate goal in mind of improving student learning and raising 
educational standards. 

Strategy for the' Center 

In our planning grant proposal, we took the position that the Center must 
involve collaborators with different areas of expertise and experiential 
background, and must be based on selected areas of need that will be consist- 
ently updated. We projected a strategy for the Center that would: 

1. Identify critical needs in schools that should be addressed by the 
Cen ter 
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2. Involve qualified practitioners, statespersons , and researchers in 
the planning of long-term programs of research and development to be 
undertaken by the Center 

3. Arrange for school-based research and development settings that 
represent the diversity of settings found in the United States 

4. Arrange for collaboration with staff members of local school dis- 
tricts, regional laboratories and other research and development 
Centers, professional associations, state education agencies, and 
other relevant educational agencies to accomplish the work of the 
Center 

5. Establish communication channels to disseminate the work of the 
Center. 



R* cognizing that a series of independent research studies within the 
mission of the Center would be unlikely to yield findings sufficiently focused 
to effect improvements in schools, we planned to emphasize programmatic re- 
searcn using personnel froti across collaborating sites. The programs of 
research would be consistent with the Center's mission and targeted to impor- 
tant needs for improving schools. Thus, from an operations perspective, our 
strategy from the beginning was to match talent and resources at the consor- 
tium sites with the tasks necessary to fulfill the Center's research goals. 

Our planning efforts were consequently focused on using literature and 
research findings, the results of our needs assessments, and the advice of 
consultants and practitioners to select needs and strategies that would pro- 
vide direction for the Center for years to come. One direction was a program 
of research on uses of tests. 

Prograa on Paea of Teata 

There are distinctly different uses of tests in education associated with 
particular groups of users. Teachers use formal and informal teats for in- 
structional decision making. School administrators use them for curriculum 
review and resource apportionment decisions. Counselors use them for advising 
students and teachers. Policy makers in education use them to monitor educa- 
tion systems and to influence educational programs. Citizens use them to 
gauge the effectiveness of schools. Yet there is very little research on 
these groups' information needs or their motives for testing and the extent to 
which their needs or intents are, or can be, met by current or future testing 
practices. * 

Thia prograa of reaearch woald exaaine the use of teata for the different 
parpcsea liated aboTe, aaking whether the needa or intenta of the client 
gxoapa are being adequately net by exiating teating practices, and diacovering 
waja to iaprore teating practlcea ao that they better aerve consomer groupa. 

A distinction has been made between internal and external testing pro- 
grams depending on locus of control— wi thin or outside the school district. 
School building or district-wide achievement-monitoring testing programs would 
be internal, while state -mandated or legislated programs and Chapter 1 testing 
would be external. Another distinction may be made between tests used to 
inform users and those used as administrative devices (e.g., to control award 
of high school diplomas or to determine grade promotion). This program of 
research would seek to investigate the uses and impacts of both internal and 
ei.temal testing intended either for informing consumers or for administrative 
intervention. 
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Inltiallj, researchers at the R & D Center could ezaalne the use of tests 
In policy. During the last five years, efforts to reform education, particu- 
larly at the state level, have increasingly employed tests and test results in 
various ways, and this use of tests in the policy sphere is a growing trend. 
For example, a 50-state survey of reform measures conducted by Edacatlon Week 
(February 6, 1985) found the following: 29 states require competency tests 
for students, and 10 other states have such a requirement under consideration; 
8 states employ a promotional "gate" test, while 3 others are considering such 
a mandate; finally, 37 states have soma sort of state assessment program, and 
6 additional states have such a program under consideration. This growing use 
of tests in the policy sphere cannot help but impact on teachers and students, 
as well as on more traditional testing programs, evaluations of students and 
standards of educational excellence.. We felt therefore, that it is imperative 
that the NIE Center document the impacts— both positive and negative, the 
costs and the benefits— associa ted with these external testing programs. 
Further, we felt that there is a pressing need to develop practical strategies 
and techniques that state departments of education and local school system^ 
can use to evaluate these programs and to make better use of the information 
they can provide. 

Another Htm of inquiry within this prograa of research would be an 
exaalnatlon of ti^stlng and standard setting practices in Australia, England, 
Ireland, Gerwiny, the Netherlands, Sweden, and Japan. There is speculation 
that the testing practices in these countries have much to offer to rciform 
efforts in the U.S. (Madaus & Greaney, 1985), yet there has never been a 
systematic and thorough study of the strengths and weaknesses, contextual 
impacts, and utility of alternative practices so that testing deficiencies in 
this country can be matched to strengths of the practices of other countries. 
In order to build on the experiences of others, we must know what they have 
been and then evaluate their potential for addressing our needs. This project 
should examine and describe current practices in the selected c tries, 
compile information about their strengths and weaknesses, investigate side 
effects and contextual idiosyncrasies of each, and then develop recommenda- 
tions for testing reform in this country. 

This program of research on uses of tests was directly linked to he 
mission of the NIE R & D Center. It focused on improving the use of formal 
and informal tests in both internally and externally controlled testing pro- 
grams, expanding our knowledge of both positive and negative impacts of test- 
ing, and developing products that educators can use for school improvement. 
It can have immediate and tangible payoff for students who may be adversely 
affected by external testing policies by informing and enlightening the" 
policy-shaping community. It can expand equality of educational opportunity 
by making important information about students more accessible to educators 
and less open to misinterpretations. It will closely examine existing prac- 
tices and lead to the development of new methods and uses of testing. 

Priority projects that were identified included: 

- Use of Tests in Policy 

- Use of Tests in Schools 

- Visiting Scholars and Practitioners Project 

- School Partnerships to Integrate and Test the Center's Products 

- Studies of Testing and Standard Setting Practices in Other Countries 
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The research on the use of tests in policy was chosen after a careful 
review of related literature. During the last five years, efforts to reform 
education, particularly at the state level, have increasingly employed tests 
and test results in various ways. First, tests have een used to inform 
policy makers about the state of education, and second, they have been used as 
administrative mechanisms to drive policy. The latter is accomplished by 
attaching important rewards or sanctions to test performance: high school 
diploma, compensatory funding, merit pay, district certification, etc. The 
use of tests in the policy sphere is a growing trend and cannot help but 
impact on teachers and students, as well as on more traditional testing pro- 
grams, evaluations of students, and standards of educational excellence. 
Therefore, we proposed to document the impacts— both positive and negative, 
the costs and the benef its— associa ted with these external testing programs. 
Further, we saw a pressing need to develop practical strategies and techniques 
that state departments of education and local school systems can use to evalu- 
ate these programs and to make better use of the information they can provide. 
This work should help to minimize the potential negative outcomes or abuses 
that can result from such programs. 

A consideration of the possible effects at the school level of using 
standardized tests suggested that one might expect effects on school organ- 
ization and on a number of school practices. Two relatively recent studies of 
the impact of more traditional testing programs indicated that such programs 
have little effect on school level organization or administrative decisions. 
Sproull and Zubrow (1981), after an intensive, small-scale study in Pennsyl- 
vania, concluded that test results from traditional school district testing 
programs were not very important to central office administrators and that 
administrators are not major users of test information. In an experimental 
study of the effects of introducing standardized testing in the schools of the 
Republic of Ireland, Kellaghan, Madaus, and Airasian (1982) found that school 
principals, when questioned about various aspects of school organization and 
practice, indicated that, at the administrative and institutional levels, the 
overall impact of a standardized testing program was slight. The findings 
extended to a wide variety of functions including: admission to school, the 
content of school report cards, streaming practices, provision for remediation 
and referral, communication practices, retention in grade, and the curriculum. 

In interpreting the negative findings cited above, one must keep in mind 
that they speak only to the effects of information from traditional school 
district testing programs on institutional organization and practice. They 
were carried out before the advent of sta te -mandated testing programs aimed at 
reforming education, and before recent efforts by school superintendents to 
use test information to drive instruction (sometimes referred to as measure- 
ment-driven instruction) or before the use of tests to continuously monitor 
student achievement (sometimes referred to as continuous achievement moni- 
toring, or CAM programs). 

The large-scale research that is available on teacher-level effects of 
standardized testing is based on surveys of teachers, most of whom had had 
considerable experience with such testing (Goslln, 1967; Beck & Stetz, 1979- 
Salmon-Cox, 1981; Kellaghan et al.,1982). These surveys show that standard- 
ized tests, while viewed favorably by teachers, were not of great relevance in 
their work. On the other hand, logic suggests that when test results carry 
with them important consequences for pupils, teachers, or schools (as is the 
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case with many externally impo^iec' testing programs), then the perception of 
their relevance might be quite different. Teacher perceptions of tesr r-^le- 
vance might also be quite different when the test results are part of a local 
measurement- driven instruction proj/jram or a CAM program. 

It seema reasonable to assume that standardized test score information 
has its most serious impact on the student. Thus, it was not surprising to 
find that most research on the effects of standardized testing has been con- 
cerned with effects on pupils. 

Goslin (1963) suggested two levels at which test information might affect 
students. The first level is the direct impact of providing students with 
additional information about his/her own abilities in the form of test scores. 
The second level of effects on students are those that result from communi- 
cating test results to other people who in turn take actions that impact on 
the student(s). 

Bloom (1969) has ergued that if the tests are understood and utilized 
properly by students and teachers, they can do much to enhance a student's 
learning as well as his/her self -concept. On the other hand, it is conceiv- 
able that learning the results from a test might adversely affect an individ- 
ual s self-concept, level of aspiration, or educational plans. Empirical 
evidence relating to the impact of providing students with test information in 
noncognitive areas is surprisingly scant. 

Part of the reason that research in this area is so sparse is the com- 
plexity of investigating the issue. Important distinctions have to be made 
between the pupil's age (test results often are not directly communicated to 
young pupils but are to secondary students); the kind or amount of information 
provided (norm- or criterion-referenced information, achievement or ability 
information); and the type of testing program involved (external test with 
important sanctions associated with the results or traditional school-based 
testing programs). The measurement of the students' self-concept is also no 
easy task, as self-concept is not a unitary trait. 

While there are no hard data available, there was considerable anecdotal 
evidence presented at the 1981 NIE-sponsored hearings on minimum competency 
testing that many students who failed a graduation Lest the first time dropped 
out of school never taking the test again. Whether or not the decision to 
drop out is directly or indirectly related to failing the graduation test is 
unknown (transcriptions and videotapes of these hearings are available from 
NIE; . 

Providing test information to students may also directly impact on thei- 
academic performance. This is the belief behind many measurement-driven 
instruction, CAM, and mastery learning programs. Further, this is an argument 
often made to justify state- level testing programs, particularly those direct- 
ly linked to promotion or graduation decisions. 

Popham, Kruse, Rankin, Sandifer, and Williams (1985) report that student 
test scores have risen dramatically in Texas, Detroit, South Carolina, and 
Maryland. In all of these locations, measurement was perceived as a catalyst 
to improve instruction. In addition, a number of people have pointed to a 
sharp decrease in the numbers of students failing minimum competency gradu- 
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ation tests as evidence of the program's success. However, alternative expla- 
nations for these gains have not been sufficiently explored. They may be due 
solely to teaching to the test. They may not generalize to other measures of 
the same construct and in fact may change the original construct the test was 
designed to measure. In all of the programs cited above, students presumably 
were made explicitly aware of their performance. Nonetheless, it is difficult 
to ascribe the gains in test scores solely to this fact, since the teachers' 
instruction presumably also changed to come in line with whatever the tests 
were measuring. 

LeMahieu's (1984) evaluation of a CAM program in the Pittsburgh Public 
Schools shed some light on hov these measurement-driven instructional programs 
work. His results ir.dicated that the program had generally positive effects 
on students' achievement as measured by test scores. He found that the CAM 
program clearly focused the attention of students and teachers on the skills 
to be measured, and that this largely accounts for the improvement in achieve- 
ment. However, LeMahieu pointed out that this focusing phenomenon also raises 
the following concerns: 

1. the routinization of instruction by some teachers who may adopt the 
objectives of the monitoring program as the sole content of instruc- 
tion in that domain 

2. a loss of residual learning outside of the CAM content 

3. as additional areas of the curriculum are added to the CAM program, 
they might begin to compete for an extremely important and limited 
resource— instructional time. In fact, Pittsburgh teachers reported 
that they took the time for supplemental instruction in math (the 
area covered by CAM) away from other subjects. LeMahieu suggests 
that these difficulties can be overcome by careful planning and wise 
management but that these dangers are real and ever present. 

Three different theoretical frameworks guided our approach to this re- 
search, namely aspects of organizational theory, information theory, and the 
multiple methods approach to research. We cannot elaborate here in any detail 
on how each of these perspectives guided the proposed research. Hence, with 
respect to each we simply give one prime example of how each guided our 
thinking. 

Sociologists of educational organizations have in recent years charac- 
terized schools as "loosely coupled" organizations in which units such as 
schools within districts and teachers within schools have a fair amount of 
autonomy in carrying out their duties. A related concept is that of teachers 
as street level bureaucrats;" that is, government employees having explicit 
responsibility for carrying out state policies, but having much autonomy in 
their day-to-day work and, burdened by many demands on their time and limited 
resources, inevitably having to engage in many accommodations in order to 
carry out their duties. These ideas provide important perspectives on schools 
as compared with more tightly structured organizations (such as many business 
organizations), but obscure the fact that schools are in fact part of an 
educational system structured in highly hierarchical fashion, with states 
having constitutional authority for education, school districts typically 
being given considerable leeway in implementing educational policy, and 
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teachers within schools having much autonomy in carrying out t^.air dav-to-dav 
responsibilities. / / 

These considerations led us, in looking at the impact of testing, to two 
important general points. One is that one must examine each of thes^ levels 
of educational organization in order to thoroughly understand the effects of 
testing. Thus, various of our hypotheses are aimed at each of these levels. 
The other is that because schools are loosely coupled organizations, with much 
room for accommodation in much of the day-to-day routine of instruction, to 
understand the role of testing we need to delve beneath official policies in 
order to see how testing affects the accommodations that teachers and students 
inevitably must make in meeting the demands of complex social organizations. 

The key idea drawn from information theory with respect to the proposed 
research program was simply that one cannot judge the value of a particular 
piece of infomation-or kind of inf orma tlon-in a vacuum. One must also look 
at the signal to noise" ratio; that is, the possible variety of competing or 
confirming information in addition to test information that may bear on a 
particular issue The importance of this point was aptly demonstrated in 
Raudenbush s (1984) meta-analysis of teacher expectancy studies which clearly 
showed that the expectancy effects of test information were greatest when 
teachers had little previous experience with students, and hence a relatively 
omall store of previous information on students which might influence their 
opinions. Though this point may seem obvious, it represented a perspective 
which has not much informed previous research on the effects of testing on 
curriculum and instruction. Thus, in looking at effects of testing on these 
broader aspects of educational organizations, we must be alert to alternative 
sources of information and influence on curriculum and public opinion regard- 
ing schooling. It also illustrated the importance of the distinction we drew 
between using tests as administrative devices versus using them to inform 
policy. When tp.sts are used as administrative devices, the signal-to-noise 
ratio of test information obviously is raised. 

The third key perspective informing our research was the multiple methods 
approach to research itself. It is well established in social research that 
the particular methods of inquiry one uses affects both what one looks for and 
what one sees. In psychological research, Campbell and Fiske (1959) are known 
for proposing their multitrai t-multimet.hod approach to construct validation 
but their general perspective is relevfmt to other forms of social research! 
including research on the effects of testing. Therefore, in looking at the 
effects of testing, we need to employ multiple methods of inquiry, looking not 
just at patterns of test score performance over time, but also at survey 
evidence, informed opinion gathered through interviews, and quasi-experimental 
evidence, as discussed in the procedures section of our full proposal. When 
results from different methods of inquiry converge, we can have confidence 
that results are not merely artifacts of one particular method of inouirv 
employed. ^ 

The primary research approaches that would be employed in the study of 
policy uses of tests were proposed to be surveys, interviews, and 
qua si-experiments. 

The program of research on uses of tests also included a proposal to 
study the uses of tests in schools. There have been several naturalistic 
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studies of test and evaluation use in schools completed over the past ten 
years (Alkin, Daillak, & White, 1979; Kennedy, Apling, & Neumann, 1980; 
Rudman et al., 1980; Radwin, 1981; Salmon-Cox, 1981; Sproull & Zubrow, 1981; 
King & Pechman, 1982). Based on these studies, it is safe to conclude that 
neither testing nor evaluation has been well integrated into the everyday 
practices of classroom teachers and school administrators. That is, the 
potential of testing and evaluation for improving student growth and develop- 
ment has not come anywhere near realization. 

When confronted with the reality of the minimal role of testing and 
evaluation in schooling, one response has been to abandon them in favor of 
less systematic, less formal means of generating information for educational 
decision making. Lor tie (1975) and Kennedy, Apling, and Neumann (1980) have 
documented occurrences of this response in schools. A side benefit of such a 
response is the elimination of misuses of testing and evaluation that have 
been reported in recent years (Holmen & Doctor, 1972; Brickell, 1976; House 
et al., 1978; Madaus, Airasian, & Kellaghan, 1980). 

To throw out testing and evaluation altogether, however, is akin to 
throwing out the baby ^ith the bath water. Benefits that may be derived from 
information generated through testing and evaluation have been widely dis- 
cussed and accepted (Stuff lebeam, et al., 1971; Kellaghan, Madaus, & Airasian, 
1982; Haney, 1984). These include: (1) identification of student needs; (2) 
guidance for selecting among known alternatives in instruction; (3) reductions 
in the influence of prejudice and impressions in making decisions affecting 
students; and (4) justification of expenditures of public funds. Without 
results of testing and evaluation programs available, school administrators 
have been found to engage regularly in personnel and program decisions (such 
as making assignments, planning in-service programs, setting goals for in- 
dividual schools, making budget allocations, and selecting program designs and 
materials) based on limited and often subjective information. 

Armed with this knowledge and an understanding about how testing and 
evaluation are perceived by school staffs, researchers need to ask how testing 
and evaluation practices can be tailored to fit the typical information needs 
of school teachers and administrators to help them make better decisions and 
hence to improve the quality of instruction offered. Can the routines just 
described be made easier for school staff members, while also being made more 
accurate, systematic, fair, and comprehensive? "is there a better way to do 
it?" is a question that may be asked of each routine activity. Research on 
that question, keeping sight of the need to make life easier for school staff, 
is the challenge to be addressed by this research project. 

The purpose of this project was to gain a better understanding of the 
information needs of teachers, principals, and other school professionals and 
how these needs are best served by improved formal and informal testing meth- 
ods, reporting and test interpretation techniques, and school-based micro- 
computer systems. 

Specifically, the objectives of this project were to conduct: 

1. descriptive case studies of student testing and evaluation practices in 
two schools and two school districts in the greater Kalamazoo, Michigan 
region; 
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2. psychophysical studies of how teachers, principals, school district 
administrators, counselors, and school board members interpret and use 
simulated test results; 

3. a feasibility study of transporting a school-based microcomputer student 
information system from the school where it has been developed (the M. L. 
King School in Pittsburgh, Pennsylvania) to another school (near 
Kalamazoo, Michigan) which has no prior experience with the system. 

teoed on the results of the final year of this project, plans for studies 
of information needs of teachers, principals, and other consumer groups for 
testing in education and how these needs can best be served, could be prepared 
for use in subsequent years of the project. In particular, practical support 
materials to aid in the reporting and interpretation of test results could be 
designed and pilot-tested. Also, experience gained from the feasibility study 
of school-based information systems will be used to design aid test student 
information systems in a variety of school settings. 

A second line of inquiry in this project was to plan to conduct psycho- 
physical studies of how teachers, principals, school district administrators, 
counselors, and school board members interpret and uee simulated test results. 
We would develop and present hypothetical test results to samples of teachers 
principals, school district administrators, counselors, and school board 
members drawn from at least ten different school districts and ask them to 
perform a varlaty of different classification tasks based on the test results. 
The stimulus test results would be systematically varied on at least three 
dimensions (namely, the title given to the test, the scale in terms of which 
results are reported, and the precision with which results are reported). 

Analyses of the ways in which people perform the classification tasks 
would provide evidence on which to base interpretations and reporting of test 
results in different ways. 

A third line of inquiry was to conduct a feasibility sLady of transport- 
ing a school-baaed microcomputer student information system from the school 
where it has been developed to another school which has no prior experience 
vUh the system. staff at the Learning R&D Center in Pittsburgh have de- 
veloped a prototype system for making necessary student information available 
to school personnel. The feasibility of transporting this system to another 
school in the greater Kalamazoo, Michigan area could be investigated to assess 
its potential for responding to the real information needs of school staff and 
the transferability of this technology. 

R&D Center staff would arrange with several school district superin- 
tendents and building principals to conduct interviews with their school staff 
about the information they currently have and use. Classroom teachers and 
building principals would investigate, through panel discussions and faculty 
meetings, how such a system could work for them and how it would have to be 
adapted to make it useful and workable. Reports from each of the schools will 
be used to describe the feasibility of using a microcomputer to better inte- 
grate testing into instruction and school improvement. 

Interviews and observations would be conducted. Inservice workshops 
would be held. And use of the system would then be discussed. A report of the 
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feasibility of using such a system to better serve the testing needs of local 
educators, with recommendations for further research and development would 
then be prepared. 

The program of research on uses of tests also included a proposal for a 
visiting scholars and practitioners project. We believe for the Center on 
Student Testing, Evaluation, and Standards to be effective, it must (1) ensure 
that its programs are responsive to the needs of schools and (2) integrate the 
products of its programs and projects into systems that will work in schools. 
The purpose of this project is to collaborate with practitioners and scholars 
to: a) assess needs in schools related to testing, evaluation, and standard 
setting; b) evaluate the relevance and practicality of the Center's contribu- 
tions from individual research and development projects; c) plan the Center's 
agenda of installation and demonstration projects; and d) set up one prototype 
school-Center partnership project aimed at helping a school to improve its 
collection and use of testing and evaluation. 

The operational framework for this project is a two-year colloquium. The 
participants would include school personnel from the Kalamazoo area. Center 
personnel, and visiting scholars and practitioners. The participants would 
collect needs data through surveys and a study of a selected pilot school, 
study the reports of the Center's projects, review relevant related research, 
assess^ the possibilities of combining the projected contributions of the 
O.nter's other projects for use in schools, engage in collaborative planning 
with one school to set up a school-partnership project, and document what is 
learned through the colloquium experience. In a very real sense this group 
would help the Center Director set the stage for the Center's development 
projects to be conducted in 1988 and 1989. Those projects are projected to 
include several school-Center partnership projects. 

The benefits from this project would be of five major types. The Center 
could gain a more concrete view of the kinds of assistance that schools need 
in the areas of testing, evaluation, and standards. Feedback from the project 
could he used to assess and improve the relevance and practicality of the 
Center's programs. One school would be assisted to assess, synthesize, and 
opera tionalize the contributions from the Center. A model plan for a school- 
university partnership project would be provided. An agenda of developmental 
school-Center partnership projects would be developed. The consulting practi- 
tioners and scholars would produce publications, based on their work in this 
project, for use by both researchers and practitioners. And, the participants 
in the colloquium would be provided a rich learning experience. 

Common criticisms of research in education are that it has had little 
influence on practice and that the timelag between invention and adoption of 
educational innovations is long (for example, see Huling, Richardson, & Hord, 
1983). Among the reasons given for this less than optimum impact are: 

- the failure to involve practitioners and the schools in the planning 
and implementation of research and development 

- the use of highly structured methods, such as experimental design, 
that do not accommodate the realities of tht classroom, school, and 
school district 

- the isolation and remoteness of university research programs from the 
setting of classrooms and schools 
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- failure to recognize a focused perspective of the schools' problems 
and their possible solutions 

- development of programs in isolation without engineering them to fit 
the context of the school or school district 

The members of our Center have had a long history of working with schools 
and school districts. While we have recognized the need for developing dis- 
crete projects with reachable goals, we also recognized the potential hazard 
of developing procedures and systems that fit the development context but not 
that of other school situations. This project would provide a main strategy 
for pulling together the Center's other projects, integrating them, and making 
them responsive to the problems of practitioners and the schools. 

The overall aim of this project is to develop and implement a strategy 
for maximizing the contributions of Center projects through collaborative 
relationships among scholars, practitioners, participating schools, and Center 
personnel. To accomplish this goal, the primary subgoals of this project 
are to involve practitioners, scholars, and staffs of the Center and a se- 
lected school to: 

1. identify needs of school personnel that relate to the mission of the 
Cen ter ; 

2. evaluate the potential contributions of the Center's research and 
development projects to address the assessed needs of selected 
schools ; 

3. create an agenda of development projects that reflects the needs of 
selected schools and provide for integrating the contributions of the 
Center's other projects into systems that can be tested in schools; 

4. develop an agenda of school -Cen ter projects to be implemented in 
years three and four of the Center's operation; 

5. set up a model for school-Center partnerships in the mission area of 
the Center. 

A secondary goal of this project would be to capitalize on the expertise 
of the participating practitioners and scholars to produce documents of gen- 
eral interest to the educational community and of interest to specific 
audiences. 



Accomplishing the above goals would greatly increase the ability of prac- 
titioners to influence the Center and increase the probable impact of the 
Center's research and development on school practice. 

Program on Hew Technologies in Testing . 

A second direction that our literature reviews, needs assessments, and 
interviews took us was toward research on new technologies in testing. Cur- 
rent reform efforts across the United States are emphasizing that education 
must be more accountable. Measures of student achievement are looked upon as 
the ultimate indicator of the success or failure of the schooling process. 
This increased attention on student testing is presenting a serious challenge 
to the education profession as to how new systems and techniques can be de- 
veloped to assess student achievement. 
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The advent of microcomputers with memory capacity and operating speeds to 
rival mainframe computers of 15 years ago has opened the possibility for major 
advancements in how we assess students. Increased capabilities in the field 
of communications also present possibilities for improving assessment strate- 
gies. Consistent with the proposed mission of this center, this program would 
identify the best strategies within testing and combine these with the latest 
innovations of technology. 

Merging technological advancements with the best In testing strategies 
can be achieved through a collaborative effort, one that includes participa- 
tion from universities, industry, and local school districts. This consor- 
tium, with the participation of industry leaders, provided a natural base for 
a major effort in developing and demonstrating new systems and techniques in 
student assessment. 

New systems would be developed on a project basis with priority need 
areas being identified from Jnput provided by school personnel and measurement 
experts. Each project would be developed in a systematic manner with detailed 
monitoring and feedback. Student data for each project would be collected 
from diverse school systems, including Dallas, Kalamazoo, and Pittsburgh. The 
projects would be enhanced by carefully designed evaluatitus from both inter- 
nal and external evaluators. The control provided through pilot testing in 
the participating school districts would assure products that are both state- 
of-the-art and practical. 

Priority projects that were identified included: 

- Adaptive Testing - computer-driven testing of students on items based 
on an existing curriculum. Each student would be tested on items 
adapted to his/her level of capability or level of instruction. 

- Data Ba sed Decision Making - Strategies would be developed and re- 
searched to find the most effective ways to display and interpret test 
data to enhance decision making. 

" "3es of Tests with Bil ingual Populations - In depth work with teachers 
to determine and respond to the unique testing problems of bilingual 
students . 

- Diagnostic Testing and In structional Management - Using adaptive test- 
ing strategies, methods would be researched to optimally iapact the 
instructional process. Students in such diverse populations as special 
education and talented/gifted could be tested for accurate placement 
within the instructional process and monitored for achievement pro- 
gress . 

Future research projects would be developed to focus on problems of 
testing with mildly handicapped populations. 

Current research in testing, item bank theory, and adaptive testing indi- 
cates a need for a well-articulated, pragmatic study of adaptive testing of 
student achievement and mastery of curriculum goals. Our proposal described a 
research project that uses current curricula and item sets in mathematics at 
the elementary level to create an integrated, microcomputer-based system for 
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adaptive testing of student achievement. The major components of the system 
included (l) a calibrated and ins true tionally indexed item bank for testing 
progress in the curriculum, and (2) a user-friendly, integrated software 
system that allowed for: 

a, the selection of tests and test items through the bank indexing and 
calibration system 

b, the production of placement and diagnostic information for use by 
teachers in assessing student progress 

c, the production of student-performance based data for instructional 
planning and management 

d, summative assessment of student progress in the curriculum 

e, two-way communication with mainframe computer systems 

The research outcomes of the proposal included data relative to the prob- 
lems of: 



1, psychometric bases of item bank indexing and calibration systems 

2, item bank indexing and calibration for use in assessing student 
achievement of curricular material 

3, measurement characteristics of tests constructed from indexed items 
and the robustness of these measures in pragmatic applications 

4, the use of adaptive testing data in conducting and assessing standard 
setting 

5, instructional refinement through adaptive testing 

6, the use of diagnostic test information in formulating strategies for 
instructional management 

With the technological breakthroughs in microcomputer design in the past 
five years, computerized adaptive testing has become a practical, as well as 
an affordable, concept. Many discussions have taken place regarding the 
potential for the use of a microcomputer-based adaptive testing system (ATS) 
in conducting student placement, diagnosing student problems, and assessing 
student achievement. The measurement issues have been discussed, and the 
technological potential has been assessed. Most of the discussion and re- 
search, however, has been hampered by the lack of an actual system to use in 
testing the hypotheses and the robustness of the assumptions. 

Beyond the need for a carefully constructed ATS, a practical void has 
existed in the realm of linking an ATS and its output to classroom instruction 
and instructional management. Our proposal was designed to fill that gap. 
The proposal was to create an ATS based on current research using existing 
curricula and existing test items in elementary school mathematics. This 
system would use an integrated software management system to create and admin- 
ister tests, provide instructional feedback relative to student progress, 
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provide Instructional management information to teachers, provide data for 
instructional standard setting, and communicate with mainframe data bases. 

The research goals of the project would be to use the constructed ATS to 
provide information relative to the problems of: 

1. the psychometric basis of item bank indexing and calibration 

2. item bank indexing and calibration for use in assessing studenc 
achievement of curricular material 

3. measurement characteristics of tests constructed from indexed item 
banks and the robustness of these measures in pragmatic applications 

4. the use of adaptive testing data in conducting and assessing standard 
setting 

5. the refinement and improvement of instructional management through 
the use of information through the ATS 

Currently, none of these areas have been investigated in the context of a 
functioning ATS. 

Related research on ATS use has been focused in three areas. The first 
has been concerned with the issues in item banking and the corresponding use 
of item response theory. The second area has been the technical character- 
istics of ATS-gecerated tests. Third has been research related to the poten- 
tial for using a microcompu ter -based ATS to improve student assessment, 
achievement, and Instructional management. 

MiUman and Arter (1984) and Wright and Bell (1984) present useful over- 
views of item banking that discuss the broad issues associated with its appli- 
cations. The most useful part of the MiUman and Arter discussion is an 
outline of the major issues involved in item banking and issues of concern for 
those attempting to create or evaluate an item bank. Wright and Bell discuss 
the mathematical and psychometric foundations of item banking in addition to 
discussing its potential. Both studies suggest the potential for using item 
banks in student assessment. They also say such banks can provide information 
for teachers and for curricular improvement. While both studies discuss the 
potential for curricular improvement, however, neither cites any actual at- 
tempts at implementation or offers any insights into how the linking between 
an ATS and instructional management would be accomplished. 

Green, Bock, Humphreys, Linn, and Reckase (1984) provide a discussion of 
the characteristics of computerized ATS systems. They address item response 
theory, dimensionality, reliability and measurement error, and validity. They 
conclude that adequate procedures exist for assessing the properties of such 
systems, but that a great deal needs to be learned about them and their poten- 
tial. Kreitzberg and Jones (1980) presented the results of an empirical study 
of a minicomputer-based ATS for a test of verbal ability. They point out that 
such systems are pragmatically feasible, that a need exists for trials of 
these systems in the field to assess student achievement rather than aptitude 
and that a need exists to investigate microcomputers as the delivery media for 
these systems. 
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Hambleton and Swaminathan (1985) suggested three important benefits of 
using item banking. First, test developers will easily be able to build tests 
to measure objectives of interest. Second, test developers will be able to 
use item banks to produce tests with the desired number of test items per 
objective. Third, when item banks contain content-valid and technically sound 
items, test quality will usually be better than what test developers could 
produce themselves . 

Hambleton, Anderson, and Murray (1983) indicated great potential for the 
use of microcomputers in classroom testing. They report possible uses of ATS 
systems for improving instruction, more accurately assessing student ability, 
providing immediate student feedback, etc. 

The foregoing examples illustrate the potential of item banking and ATS 
for improving classroom instruction. However, before this potential can be 
completely realized, certain needs must be met. For example, several ques- 
tions about the theory underlying ATS systems and their properties must be 
answered. In addition, fully operational versions of such systems in class- 
rooms settings must be created so that the explicit links between implementing 
an ATS and using the results to improve instructional management can be deter- 
mined and tested in a natural setting. 

This proposed research had direct applicability to the Center's mission 
and provides a vehicle for the investigation of other problems germane to the 
Center. It also crossed into the research domains of other proposed centers 
and thus provides a means for collaborative research. It was first and fore- 
most directly applicable to the Center^s objective of investigating new tech- 
nology and its applicability in the classroom to improve measurement and 
learning. The proposal provided for a system that incorporates state-of-the- 
art technology with the cumulative research in ATS theory to give a 
curriculum-based system with research applications in standard setting, stu- 
dent testing;, evaluation, and psychometric theory. It also offered the oppor- 
tunity for researchers to gather a wealth of information from field tests in 
actual school settings and establish item banks in coutent areas and grades 
where improved testing is greatly needed. This was particularly relevant 
givea the current Interest in elementary mathematics instruction. 

Finally, this project offered a link with the investigation of instruct- 
ion and the coordination of instruction and testing by its ties to existing 
curricula and test items in use in school systems. These linkages are also 
established by the indexing of the item bank according to the instructional 
objectives of the curricula. In line with the Center's mandate to collaborate 
with the other proposed centers, this project offered ideal ways for linking 
with those centers that deal with instruction and instructional improvement 
and centers dealing with teacher education. 

Prograa On Standard Setting s 

The third direction that our review of literature, needs assessments, and 
discussions took us was toward a program of research on standard-setting. This 
program was keyed to understanding the nature of educational standards held by 
our citizenry, developing methods that are capable of accurately reflecting 
societal expectations, and examining the consequences of standards on educa- 
tional processes and outcomes. Today there is considerable public concern 
about the standards of our schools. Standards evolve from societal needs and 
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are then shaped by what is taught and eventually by the manner in which we 
judge what has been learned, e.g., through testing. Clearly, standards do not 
exist in a vacuum. They affect and are affected by what is taught and what is 
found by way of testing or evaluation. Given the central role that standards 
play, there is a need for systematic inquiry on standards so that the comple- 
mentary forces of testing and evaluation and of teaching and administration 
contribute to improved school practices. 

A major line of inquiry on standards that we planned to undertake was 
geared to understanding the process by which standards evolve. Evidence of 
educational standards can be seen from classroom grading practices to state 
requirements for high school graduation. Yet little is known of the social, 
cultural, political, and economic factors that contribute to the establishment 
of standards or the role played by the various stakeholder groups when stan- 
dards are set or accepted and used. Further, the audience education serves is 
diverse, and studies are needed to examine the relationship of standards to 
the values and needs of distinct populations. Utilizing case studies at the 
level of teachers and the local schools, and at district and state levels, 
could lead to an understanding of how standards are formed. Investigations 
were also planned to address standards and the process of their formulation 
across what are judged to be effective and noneffective schools. 

A second area for investigation addressed the methodology of standard 
setting. While the recent past has witnessed a great many reported studies of 
the characteristics of standard-setting techniques, research to date has been 
narrowly focuaed. There is a need for research on issues such as the effect 
of directions on resulting standards, clear definition of the components 
required if equitable and acceptable standards are to be set, examination of 
the appropriateness of fixed standards for different populations, exploration 
of alternate methodologies for standard setting, and issues of test instruc- 
tional validity tied to standard setting on tests with different purposes 
(e.g., CRTs vs. NRTs). 

Beyond these properties of methods, technical questions remain regarding 
such issues as test length and composition, appropriateness of various stan- 
dard errors for deriving accurate cut points, the utility of multiple thres- 
holds, and the extent to which test dimensionality, defined by either content 
domain or test structure factors, confounds standards. If fair and equitable 
standards are to be established, these are among the many issues requiring 
attention. 

Research was also planned to address the question of the consequences of 
standards on educational practices and outcomes. While there has been consid- 
erable speculation and debate, this area has not been systematically studied. 
A few of the questions to be addressed include: Do standards prescribed as 
minimal become maximal? What are the consequences of teacher/test classifica- 
tion discrepancies? How can test result information be best reported to 
facilitate use? What form of information is most useful for different audi- 
ences? What are the shifts in standards over time? In what ways do standards 
affect school processes? 

While questions relating to standards are to be addressed at a number of 
levels, our focus was largely at the local/ community level, for it is at this 
level that test and evaluation results must be responsive if school learning 
is to be maximized. 



Priority projects that were identified included: 



- The Process of Standard Setting in Effective and Ineffective Schools 

- Study of Methods and Techniques for Setting Standards 

- Competency Testing and its Impact on Educational Standards 

- Methods and Impact of Reporting Test Results to State and Local 
agencies 

That American schools have failed to set and maintain proper standards 
for what Is learned and how learning Is judged is a proposition that enjoys 
virtually unlveraal national consensus. General reports of the schools' 
fallings are common (see, for example, Boyer, 1983; Goodlad, 1983; National 
Research Council, 1977; National Commission on Excellence in Education, 1983; 
Sizer, 1984), as are reports dealing with specific curricular areas (e.g.. 
College Entrance Examination Board, 1977; Chall, 1977). Proposals for remedy- 
ing this problem have also appeared (e.g., Adler, 1982; Resnlck & Resnick. 
1985). 



Despite this surfeit of information, the question of how standards actu- 
ally evolve at the local school district level remains for the most part 
unanswered. If a school district determined to respond to the national cry 
for accountability and establish m v and presumably higher standards, what are 
the ways it should go about that task? Even supposing that a district had 
complete and accurate Information on what little the research has shown on 
these matters, how could it productively use that information? 

There are multiple factors Involved in reaching appropriate decisions and 
many of these factors are in conflict. For example, schools seem to be ex- 
pected to share a common national set of standards; indeed, much of the lit- 
erature mentioned in the preceding paragraph implies that national standards 
are imperative. Yet politically, schools are under local control. How can a 
standard-setting process take account of this bifurcation in power? The 
proposal that there be national standards presupposes that there is a viable 
mode for determining what they should be. But can that be so? The nation's 
experience with, and our own understanding of, cultural and value diversity 
suggest otherwise. "National standards" also suggest some common core of 
knowledge and skills that all should possess; however, for a century it has 
been an article of faith among educators that good teaching caters to and 
capitalizes upon individual differences. The national desire to maintain 
equal opportunity for all can be taken to imply that everyone should be pro- 
vided a college preparatory curriculum so as not to deny the student the 
option of college. On the other hand, tracking practices and the demands for 
adequate vocational education seem to preclude such opportunity for many. 
Standards are often said to be set by textbook publishers, who may have their 
own reasons for determining book content. To the extent that this is true, 
what flexibility adheres to the local district, especially in states that 
regulate textbook adoption? 

It seems clear that standard setting is not simply a matter of deciding 
what to do and doing it or deciding to do it better and doing that. The 
standard-setting process historically seems to have been less a matter of 
making rational, research-based decisions than making compromises, adapta- 
tions, and accommodations with a variety of standard-setting forces that seem 



almost to have a life of their own, one well beyond the capability of the 
typical local district to control. And it seems likely that efforts to alter 
(hopefully to raise) standards will not be successful unless a great deal more 
is understood about how that process occurs "naturally." It is even question- 
able whether standard setting has been a process in which local districts have 
been consciously involved. They may simply have been forced into compliance 
with standards shaped by outside forces. 

The purpose of one project that we proposed was to provide insights into 
the question of how standards actually evolve at the local school district 
level. It seemed clear that if effective approaches are to be taken toward 
3etting new standards, it is imperative to know something about how the pro- 
cess occurs in a real-life setting. 

The proposed study has three major objectives: 

1. To study in situ several school districts with a view to describing 
how standard setting occurs in each. This objective will result in 
one or more (probably two) case studies that portray the standard- 
setting process (or lack of it) as reconstructed by "insiders," that 
is, an enlc view, as opposed to the etlc (or outsider's) view. 

2. To test, as part of the process of carrying out the first objective, 
assertions currently found in the literature about the standard- 
setting process, as well as current recommendations for its improve- 
ment. Can evidence be uncovered to suggest that existing research 
descriptions, which are essentially generalizations putatively "true" 
of a statistically average district, hold true in particular and 
concrete settings? 

3. To make recommendations of two different sorts as an output: 

a. Considering the case iStudies carried out in this project how can 
other case studies be mounted, perhaps purposefully selected to 
contrast with those already completed along dimensions of factors 
which the initial studies suggest are important? What sugges- 
tions emerge from the present study to improve the methodology to 
be employed? 

b. Considering what is suggested by the case studies about the 
degree to which factors described in conventional studies work 
themselves out at the local level, how can future scientific 
studies of standard setting be more adequately grounded? How can 
the methodology be altered to expose the unique adaptations and 
accommodations to local conditions? 

A second project on standard-setting was proposed to address a very 
different gap in the literature, using a very different research strategy. 
A review by Berk (1985) revealed that no fewev than 30 methods have surfaced 
and been used within the recent past to set standards for test performance. 
For the interested reader a description of the three most commonly used 
methods (Angoff, Ebel, and Contrasting Groups) is provided below. 



The Angoff, Ebel, and Contrasting Groups Procedures are based on expert 
judges' assessments with respect to the expected performance of students. The 
methods differ in terms, of the specific factors rated. For these methods, 
both Che judgments made and the standards derived are independent of the 
actual performance of students on the tests. 

Angoff method; For each test item, raters estimate the probability (on a 
scale of 0 to 100) that a minimally competent student will know the 
correct answer to the item (Angoff, 1971). In essence, the judges esti- 
mate the difficulty level of an item, referencing a hypothetical groun of 
individuals that would be judged minimally competent. To obtain the 
overall standard, probabilities assigned by a judge are summed, then 
averaged over judges to yield the passing score. This standard repre- 
sents the estimated mean total score for a group of minimally competent 
individuals. 

Ebel method; Judges make three judgments (Ebel, 1979). First, judges 
rate each test item on two separate dimensions: level of difficulty 
(easy, medium, or hard), and degree of relevance (essential, important, 
acceptable, or questionable). Then a judge indicates the proportion of 
items to be answered correctly for each difficulty and relevance configu- 
ration, e.g., easy items that are important. To derive the standard, 
each item is assigned to its appropriate cell based on the judges' rat- 
ings. The percentage passing judgment for a c^ll is then multiplied by 
the number of items in that cell, and these products are summed over all 
cells to obtain the passing score for a judge. Passing scores are then 
averaged over judges to obtain the passing score for the test. 



Contrast ing Groups method; A teacher classifies a student into one of 
two groups. Competent or Non-Compe tent, relative to the content being 
assessed (Livingston & Zelcky, 1983). Based upon these group membership 
classifications, and the actual test scores of these students, a standard 
is derived, using statistical likelihood-ratio procedures which minimize 
the probability of misclassif ica tion of students into groups. There are 
several variants in the specific statistical procedures available. 
Choice of a procedure is dependent upon the population distribution 
shapes and relative variances of the two groups' test scores. 

The methodology of standard setting has become the most researched topic 
in the criterion-referenced measurement literature. A summary of the findings 
from this literature follows: 

1. Two general classes of procedures for setting test standards have 
emerged and now hold center stage; judgmental item review-based 
methods, referred to as "Continuum Models" and empirical 
performance-based methods, referred to as "State Models" (Jaeeer. 
1976; Meskauskas, 1976). 

Within the judgmental item review class are those methods such as the 
Angoff, modified Angoff, Ebel, Jaeger (1978), and Nedelsky (1954) 
procedures. The use of such methods requires that raters, those 
persons charged with determining the performance standard, examine 
the actual test items and offer opinions as to how students are 
likely to perform and/or judge the item's relevance. The empirical 



performance methodt:, e.g., Contrasting Groups, Borderline Group, and 
Criterion Groups, derive a standard by examining statistically how 
persons tested av-itually perform in relation to how they were expected 
to perform given teacher nomination. 

2. Different methods, regardless of their class, produce different stan- 
dards (Andrew & Hecht, 1976; Poggio, Glasnapp, & Eros, 1981, 1982. 
1983). 

This finding has been reported consistently. Insofar as different 
methods ask different criterion questions of the standard setters, 
this is not an altogether surprising result. 

3. The level of the standard produced by a method in comparison to 
standards resulting from other methods can be fairly well predicted 
(Berk, 1984; Kottler, 1980; Poggio & Glasnapp, 1981, 1982; Skakun & 
Kling, 1980). 

The literature is consistent on this point and has demonstrated that 
regardless of content tested, grade of students being tested, affili- 
ation of the group setting the standard, or the nature of the deci- 
sion to be made, a method can be expected to result in a standard 
systematically different from that of other methods. The reasons for 
such trends are not altogether clear at this time. 

4. Regardless of the method used, the standard derived is susceptible to 
both measurement and sampling error (Berk, 1976; Hambloton, 1984; 
Millman, 1973; Poggio, 1984). 

All methods require judgments to be made, and the process of judging 
is fallible. Further, it has been shown that the relationship of 
judges to group membership (e.g., teachers, administrators, policy 
makers, parents) is related to the level of the resulting standard. 

5. In most applications performance is measured on a continuous scale, 
and to guard against misclassif ica tion the standard needs to be 
adjusted by the standard error of the statistic and/or measurement 
(Livingston & Zieky, 1983; Macready & Dayton, 1980; Poggio, 1984). 

This process is related to if4 above and is problematic insofar as it 
introduces another need for judgment in the absence of accepted 
guidelines • 

6. Introducing normative information to the standard setting process 
serves to improve the psychometric cnaracteris tics of the method. 

Permitting iteration of the judgment activity, allowing discussion 
among standard setters, and providing item normative data all have 
oeen shown to improve the reliability of the standard -set ting method; 
however, the actual level of the standard most often goes unchanged. 

The research reported to date has been thorough in documenting the char- 
acteristics of and similarities among methods. Yet there remain numerous 
issues requiring attention. If fair and equitable standards are to be 
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realized, further research that can assist practitioners is acutely needed. 
One need only read the reports from different locales detailing how cut scores 
were established to be astonished at the diversity of methods being used or 
the diversity with which the same method is used. Study of these efforts 
convinces us that although the users ware well intended, the research commu- 
nity has failed to be clear, precise, and thorough in presenting the methods 
and techniques for setting standards. Further, although the topic has been 
researched for more than a decade, little has been accomplished in terms of 
exploring alternative methods or giving consideration to the validity of 
existing methods. 

The research that we proposed has the following objectives: 

1. to evaluate the validity of the explicit assumptions of commonly used 
standard-setting methods 

2. to develop and test alternate standard-setting methods 
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to examine psychometric characteristics of methods and tests neces- 
sary to yield equitable cut scores 

4. to prepare a user's handbook on standard setting 

Progran oa Paychoaetrlc Theory and i^)pllcatlons 

The final direction that resulted from our literature reviews, needs 
assessments, and discussions centered on the application of psychometric 
theory of testing problems in schools. 

f ^J^^a^^^^^^^ P'^oPosed program of work is on two technical advances 

of the 1970s and 1980s: item response theory (IRT) and criterionref erenced 
testing (CRT) (Hambleton & Swaminathan, 1985; Berk, 1984). The first is a 
methodological advance that can be applied to all types of testing instruments 
and data. The second is an alternative to norm-referenced testing and pro- 
vides a basis for looking at students in relation to standards. Both advances 
are central to what is called modem measurement and have considerable poten- 
tial for improving testing practices. 

We noted first that criterion-referenced tests have more potential for 
successfully integrating teaching, instruction, and assessment than norm- 
referenced tests. Also, far less is known about the proper construction and 
uses of criterion-referenced tests than norm-referenced tests. The latter 
have been studied since about 1910; the former were only introduced in 1969 
(Hambleton, 1982). Second, in view of the potential of item response theory 
for solving a wide variety of testing problems (Hambleton, 1983), emphasis on 
this general line of research seemed highly appropriate and consistent with 
the main direction for testing research today. 

Item response theory provided a promising framework for the study of many 
testing problems, including item bias, test development, individually tailored 
or adaptive tests, and test score equating. If it can be determined that one 
or more ifem response models fit criterion-referenced test data, these models 
will be useful in helping to address several unique problems that arise in 
criterion-referenced measurement (CRM): (a) optimal item selection to maxi- 
mize the decision-making capabilities of short CRTs, (b) the need for cali- 



bra ted banks of test items where item statistics are not student group depend- 
ent, and (c) the development of scales for reporting achievement growth. There 
are also some non-unique problems that arise in criterion-referenced measure- 
ment that will require investigation prior to the full implementation of item 
respouse models in CRM: (a) assessing model-data fit, (b) estimating model 
parameters with short test and/or small samples of examines, (c) model selec- 
tion to solve particular measurement problems, (d) development of new IRT 
models with more diagnostic capabilities (Embretson, i985a), and (e) produc- 
tion of developmental scales (e.g. Bergan, 1984). 

Criterion-referenced testing technology probably has received even more 
attention from researchers over the last. 15 years than item response theory. 
Not surprisingly then, there are presently fewer major problems. There ap- 
pears to be sufficient knowledge available today to build reliable and valid 
criterion-referenced tests for local schools. Still, important psychometric 
work remains to be done. Our work focused on problems such as soecifying 
instructional objectives (this needs to be done well, because objectives serve 
as the "targets" for instruction, ^nd they are central in test development), 
determining test lengths (classical methods are not applicable), assessing 
test score and decision validity, and reporting test score information. 

Over the last 15 years, much of psychometric research has been directly 
addressed to real testing problems that arise in schools. We aim to continue 
that tradition by developing new models and procedures to solve practical 
measurement problems that were identified by our needs assessments and by our 
own observations of testing problems being presently faced by school 
personnel. 

The priority projects for the Program on Psychometric theory included: 

- Solving Criterion-Reference Measurement Problems with Item Response 
Models ^ 

- Study of Residual Analysis in Test Design 

- Advances in Criterion-Reference Test Score Reporting 

- Patterns of Item Response 

The overall goal of our psychometric research thrust is to enhance the 
usefulness of criterion-referenced tests to address several problems associ- 
ated with criterion-referenced tes ts— choosing test items, selecting test 
lengths, and adaptively administering tests. These problems are to be ad- 
dressed using models and procedures within an item response theory framework. 
In view of the newness of the technology and its infrequent application to 
criterion-rijferenced test data, several methodological issues were addressed 
as part of the research project planning: 

1* assessment of model fit 

2 . es tima tion of parame ters 

3. equating test forms 

4. assessment of item and test bias 



Depending on the results of this work, the development of some new IRT 
models may be necessary. 



The research project is organized around two related components: CRT 
methodological studies and IRT methodological studies. 

This program provides one important theoretical and methodological base 
for all of the Center's research and development work. In addition to con- 
ducting their own -rojects, it is planned that Dr;. Hambleton and Swaminathan 
will work with the ether three programs. 



ERIC 



35 



III. The Future of Educational Research and Developnent in Student Testing « 
Evaluation, and Standard Setting 



The purpose of this paper is to describe, in a non- technical manner, the 
work needed, over the next five years, to accomplish desirable goals in the 
area of student testing, evaluation, and standard setting. This paper is the 
result of a cooperative planning effort undertaken by specialists in student 
testing, evaluation, and standard setting at Boston College, the Dallas 
Independent School District, the University of Kansas, the University of 
Massachusetts, and Western Michigan University, The paper is intended to. 
summarize very briefly where these experts see research and development 
heading in the near future and how it should impact education. The views 
presented here are based on extensive literature reviews, interviews, a survey 
of key informants, and discussions among experts at the cooperating 
institutions, ^ ^ 

Trying to predict the future is problematical at best, however, it is 
possible to project how present developments in testing will affect education 
in general at least for the immediate future. This paper addresses futures in 
the use of tests, in microcomputer-based adaptive testing, in standard set- 
ting, and in psychometric theory. 

The Use of Tests 



The future of research and development on uses of tests should build on 
past efforts and on the projects described in the WMU Center proposal. There 
is a substantial body of literature on the uses of tests in educational set- 
tings that is not well integrated and which has not been analyzed to establish 
principles for educational testing that could be used to enhance the utility, 
feasibility, propriety, and technical quality of testing in any educational 
setting. The long-terra goal of a center on student testing should be to 
establish a firm knowledge base about educational testing that is then trans- 
lated into operational guidelines for educators and those who use information 
about student status and development. 

The future of research and development on uses of tests should be direct- 
ed toward answering the following questions: 

1. What organizational forms for testing have been found to be most use- 
ful, feasible, proper, and technically sound for each of these dif- 
ferent consumer groups: teachers, school building specialists, 
principals, superintendents, counselors, school boards, state depart- 
ments of education, legislators, parents, the non-parent community, 
and policy makers at the national level? Does the success of dif- 
ferent organizational forms vary systematically by educational level 
or within consumer group? 



These projects include: The use of tests in policy, studies of typical 
classroom, school, and district uses of testing, examination of testing and 
standard-setting practices in other countries, a visiting scholars and 
practitioners project, and school partnerships. 
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2. What are the information needs of each of the above listed consumer 
groups and to what extent are these needs being met by existing 
methods of testing? What needs for information are not being met by 
existing methods of testing? What alternative approaches to testing 
are likely to address each unmet need, and how do they work when 
tried? 

The long-terra objectives for research and development on the uses of 
tests should be firmly grounded on past and present research, on reliable and 
valid data on consumer information needs, and on creative design and prototype 
testing efforts. Once new approaches have been tested, they should be pack- 
aged for dissemination, and demonstration projects should be arranged. 
Specifically, researchers should: 

1. Conduct an exhaustive search for studies of testing used for 
different purposes, with different consumer audiences, and in dif- 
ferent settings. These studies should inform the research staff 
about what has been tried and about comparative strengths and weak- 
nesses of past practices. These studies should be analyzed for 
patterns of practice that may form the basis for a set of principles 
that would guide future work on the use of testing. 

2. Conduct an exhaustive search for studies of information needs of 
different consumer audiences and of the extent to which they have 
been addressed by available testing methods. These studies should be 
analyzed for patterns of needs and effective testing methods for 
reducing these needs. The analysis would form the basis for speci- 
fying needs that remain. 

3. Conduct input evaluation studies aimed at developing alternative 
strategies for addressing each unmet need. These studies should be 
aimed at preparing effective, feasible, proper, and technically sound 
approaches to meeting the needs of educational consumer groups. 
These approaches should be systematically tested and the results made 
available for public discussion and critique. 

4. Search for and develop new approaches and support materials for any 
n^w approaches to testing that are found to be effective, new means 
of achieving previously unmet and important needs for identified 
consumers of information about students. 

5. Arrange for informing key actors in education about new approaches 
with the goal of achieving widespread adoption and use of them in 
American education. 

During the last five years, efforts to reform education, particularly at 
the state level, have increasingly employed tests and test results in various 
ways. And this use of tests in the policy sphere is a growing trend. For 
example, a 50- state survey of reform measures conducted by EDUCATION WEEK 
(2/6/85) found the following: 29 states require competency tests for 
students, and 10 other states have such a requirement under consideration; 15 
states require an exit test for graduation, 4 additional states have such a 
measure under considera tioa; 8 states employ a promotional "gates" test, while 
3 others are considering such a mandate; finally, 37 states have some sort of 
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state assessment program, and 6 additional states have such a program under 
consideration. In addition to these programs, there are merit pay programs, 
compensatory funding programs, and teacher testing programs all of which 
involve the use of teat results in reform efforts. 

This growing use of tests in the policy sphere cannot but impact on 
teachers and students, as well as on more traditional testing programs, evalu- 
ations of students, and standards of educational excellence. Therefore, it is 
imperative that the NIE Center, as part of its future work, document the 
impacts— both positive and negative, the costs and the benef i ts— associa ted. 
with these external testing programs. Further, we feel there is a pressing 
need to develop practical strategies and techniques that state departments of 
education and local school systems can use to evaluate these programs and to 
make better use of the information they can provide. 

The internal use of teats in schools is equally problematical. There is 
a convincing body of literature that tells us that millions of dollars are 
spent on school testing programs that provide information that is frequently 
unused. The student information needs of local consumers, such as teachers, 
principals, superintendents, and parents are not well known, and it is dif- 
ficult to try to compare information generation (through testing) with infor- 
mation needs. Research on the uses of tests in schools must provide the 
testing industry with knowledge about consumer behavior if the utility of 
testing for improving student development is to be improved. New reporting 
systems of perhaps different information than has been traditionally reported 
potentially can move testing from the periphery to the mainstream of teaching 
and learning. 

Many of the administrative uses of tests have historical counterparts 
either in this country or abroad. Further, the use of tests as a certifica- 
tion mechanism is widespread in Europe. Therefore, the Center should identify 
and analyze past uses of tests in policy and in schools both in this country 
and abroad. This historical analysis would gather evidence on how these 
programs evolved, how they fit into the structure of the particular educa- 
tional system, and what their positive and negative effects were on teaching 
and learning. 

Mlcroco^uter-Based Adaptive Tea ting 

Microcomputer-based adaptive testing offers two immediate avenues for 
considering promising applications. The first comes in the consideration of 
pragmatic testing concerns. Here such a system has immediate applications in 
a number of areas that are described below. The second avenue comes from the 
potential of the system to contribute to research in instruction and the role 
of testing in refining knowledge about instruction. Both will be able to be 
pursued upon the successful completion of research and development on micro- 
computer-based adaptive testing, and both offer high potential for realizing 
great dividends. 

A system to be developed by the Center would provide microcomputer-based 
software which allows for the selection of items from an indexed bank for 
adaptive testing. The system would have links to an instructional hierarchy 
which would help teachers with the selection of instructional approaches with 
individual students or classes through the testing output. These two features 
of the system provide its luture potential. 
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A severe limitation to repeated testing in classrooms is the high cost, 
both in terras of time and money. Furthermore, to be useful, repeated testing 
requires rapid feedback to the teacher and student. The practical advantage 
of the proposed system is that the cost of the system is minimal when spread 
over the large number of students that can access it and the increased testing 
throughput that becomes possible. Repeated testing becomes feasible under 
either criterion. When considered in light of testing time, the testing can 
be accomplished rapidly with the use of adaptive item selection and the rela- 
tively fast response times of a state-of-the-art microcomputer. Furthermore, 
the feedback for the student and teacher comes in minutes and seconds instead 
of weeks and days. The feedback can be instantly tailored for any size group 
of students or the results for a large group can be used to form more effec- 
tive subgroups for instructional purposes. Finally, the reduced testing time 
makes increased use of the system possible without an undue detraction from 
ins true tiona 1 time • 



Another important practical advantage of the system is the ultimate 
potential for its easy adaptation for use with new technologies such as video 
disc or expanded-capability terminals. A video disc-based item delivery 
system could be efficiently and effectively merged with the system proposed, 
as could more versatile graphics terminals. Systems such as these offer great 
potential for employing new item form and testing situations. 

From the research perspective, the system offers many possibilities. A 
proposed initial base for the system is mathematics. Mathematics was chosen 
because of the large number of existing curricula and satisfactory item pools. 
The instructional linkage with these existing materials can be straightforward 
and immediately usable in the classroom. 

The research potential with this existing system will be great. As one 
example, research in mathematics instruction and testing has shown that the 
options a student selects in answering a problem give insight into his or her 
conceptual approach to the task. Knowledge of the responses to the items can 
be easily tracked for students and groups of students allowing teachers and 
mathematicians to analyze the relationship between different instructional 
approaches and the degree to which tasks are mastered. 

Greater potential comes in utilizing the testing capabilities of micro- 
computer-based adaptive testing in deciphering the hierarchical structures of 
more complex skills such as reading. The field of reading research has been 
plagued by the inability to determine effective hierarchical structures fov 
teaching reading. The literature is rife with studies which directly contra- 
dict each other. The advantages offered by the proposed system in determining 
effective structures and the concomitant teaching methodologies are immense. 
The possibility of immediate testing and of having an instructional analysis 
system which can instantly match students and their known characteristics with 
their performance under different instructional systems opens tremendous 
opportunities to advance reading research. 

In short, the long term potential of microcomputer-based adaptive testing 
is unlimited in terms of both immediate practical application and future 
advances in research utilizing testing in the analysis of effective instruc- 
tion. 
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Standards Setting 
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The common thread that runs through the future of research and develop- 
ment m standard-setting is an understanding of how standards evolve and take 
hold in society, and an examination of methods that yield equitable test 
performance standards. Further, the research programs to which we are com- 
mitted are, for the most part, tied to providing information for school-based 
practitioners. We are committed to a mission that views testing and evalua- 
tion as important elements in the teaching and learning process, but recog- 
nizes testing and evaluation as tools in this process. 

Through our intended research agenda we plan to undertake programs that 

enhance the fair and equitable use of tests at the local level. Thrusts that 

we envision for research and development on standard setting over the next 
five years are: 

1. Standards - Establishing Passing Scores 

Based on our work and others"*, we do not believe "truth" will be 
discovered in this arena. That is, a discovery of the method to 
yield the passing score is quixotic. Performance distributions are 
continuous and unimodal. The goal of a standard setting method is to 
provide an objective, reliable, and valid procedure for securing the 
values of judges in such a way that the range of possible passing 
scores is narrowed. From this restricted boundary, discussion lead- 
ing to consensus can occur. Recognizing fallibility as a premise or 
precondition, inquiries should be conducted ir such areas as: 

a. validity characteristics of a host of available methods; 

b. defining other methods - we are particularly interested in com- 
bining concerns tied to curricular and instructional validity to 
setting the cut score, and exploring scaling methods (Thurstone, 
Guttman) ; 

c. alternate standards configurations, e.g., total score-versus 
objective passing plus total score, test dimensionality and test 
length and the accuracy of classification, student population 
characteristics (e.g. handicapped) and the usefulness of 
information; 

d . inquiry into teacher-versus- tes t classif ica tion , examining 
sources of discrepancy, consequences, test use, test credibility, 
test cognitive dissonance, etc.; 

e. possibility of local standards versus imposed standards (SEA) 
tied to test use, impact, and policy; 

f. issues of equity linked to students close to the passing score, 
potentially broadening discrimination into 3 categories (at 
least), e.g., failers, remedials, and passers; and, 

g. isolation and documentation of factors that effect setting test 
performance tandards. 
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2. Standards - Impact of Testing and Evaluation on Societal Expectations 
and Implementation 



The research agenda in this aiea would be less likely geared to 
experimental/quasi-experimental s tudies. Historical as well as 
naturalistic forms of inquiry are likely to predominate. Of interest 
would be: 

a. inquiry on the political aspects and realities of evaluation 
users; 

b. questions of believabili ty , importances nd use of evaluation and 
testing products by key decision make s; and, 

c. characteristics of processes, products, and deliverables that 
effect standards formation, selection, adoption, and incorpora- 
tion. 

3. Curriculum-Test Match 

The issues here are quickly becoming apparent. Seminal studies are 
needed. Findings need to be communicated to users. Further, there 
is a need to create, then establish, designs that evaluate fit/match. 
Studies being considered include: 

a. impact on standards and adequate adjustments; 

b. extent of content taught but not tested; 

c. evaluating school effectiveness in consideration of conditions 
imposed by curriculum- test match. 

Psychometric Theory 

Presently, we can see at least three prominent directions for the 
research: 

1. criterion-referenced testing methodology, 

2. computers and testing, 

3. cognitive theory and psychome tries. 

A very brief description of each research direction follows: 

1» Criterion-Referenced Testing Methodology 

Our research would have three principal objectives: 

a. to develop some new methods for determining criterion-referenced 
test lengths. Of main interest would be a method that involves 
practitioners using the computer to simulate realistic test 
results so as to empirically investigate the effects of a number 
of factors (i.e. , cut-off score, score distribution) prior to 
choosing a test length. 
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b. to investigate the use of optimal item selection on the decision- 
making effectiveness of tests, and the consequences of optimal 
item selection on content validity and on the motivational and 
psychological influence of statistically similar test items; 

c. to design and field test several new methods for assessing 
criterion-referenced test score validity. Of special interest 
would be decision- theoretic approaches involving uew functions 
and consideration of problems in choosing samples. The results 
from chis research would include some specific guidelines for 
validating criterion-referenced tests which are consistv^nt with 
the 1985 APA/ AERA/NCME Test Standards . 

The results from the above three research studies, and related studies, 
can be of considerable value to school and state testing personnel who have 
the task of building technically sound and defensible cri terion- referenced 
tests. The research studies should be carried out in a coordinated way with 
participating school districts and state departments, so that the final re- 
sults will be understandable and useful to these important users of criterion- 
referenced tests. 

2. Computers and Testing 

We predict that in the coming years, the present computer revolution in 
this country will greatly influence the ways in which educational and psycho- 
logical tests are developed, administered, scored, and interpreted. To date, 
the impact of computers on testing practices has been limited to the uses of 
computers in item banking and adaptive t--^ting with multiple-choice test 
questions. 

The principal goal of our research prcg-a^n would be tc enhance the valid- 
ity of exam scores and associated decisions by effectively using maia-frame 
and micro-computers. Specifically, our research would center on the use of 
free-response questions, video-disc techno ogy, sequencing problems, new 
scoring formats, and other testing innovativ>ns that can be addressed with 
computer technology. To date, alicost na research along the general lines 
described above has been conducted. 

3 • Cognitive Theory and Psycht /aett^ lea 

One of the frontier areas~c,f pr^yt V.--Ttv \c3 is the merger between modern 
cognitive theory and psychome tries. ^^Idiin fjtie framework of cognitive theory, 
uew important variables that influ^iuce learning and retention are being iden- 
tified. These variables need to be fully defined and measured, and validity 
studies must be carried out. Also, new psychometric models, such as some of 
the new multi-cognitive componenti: ir7.odels beiug developed and studied by 
Fischer , Embretson, and others, must be furt:her developed. Our principal 
goal would be to build on the existing research by developing new psychometric 
models based on some of our other item response aodel research to facilitate 
development and analysis of theory» based tests, and by demonstrating the 
relevance of these new psychometric models and tests in educational settings. 
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In summary, research and development at a National Center on student 
testing, evaluation, and standard setting should be building a knowledge base 
in important areas where little is currently known. It should translate this 
new knowledge into workable models that can be used in classrooms, schools and 
school districts, and for state and national level reform. It should pioneer 
new advances which may not have direct impact on educational practices in the 
short term, but would expand the research and development landscape for future 
generations. And, school districts should be directly involved in designing, 
conducting, and applying the needed research and development. 
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Perscmnel Distribution Table 



Ca tegor 7 



Minorities | Voaen J Directors | Core Staff | Gcmsultant 



Peter Afrasian 
Cordelia Alexander 

Gilbert Austin 

Mary Anne Bunda 

Judith Burry 

William Cooley 

William Denton 

Esther Diamond 

Susan Embretson 

Arnold Gallegos 

Douglas Glassnap 

Maria Lulsa Gonzalez 

Robert Grobe 

Egon Guba 

Ron Hambleton 

Walter Haney 

Thomas J. Helry 

Grace Iverson 

Richard Jaeger 

Yvonna Lincoln 

Larry Ludlow 

George Madaus 

Samuel Mayo 

Robert Mendro 

Jack Merwln 

Edward Meyen 
M. David Miller 

Napoleon Mitchell 
George H. Olson 
Joseph Pedulla 
John Pogglo 
Sri Kanta Rao 
Thomas Ryan 
John Sandberg 
James Sanders 
Michael Scrlven 
Daniel Stufflebeam 
Harlharan Swamlnathan 
Lyke Thompson 
Carol Kehr Tittle 
Robert Travers 
Ralph W. Tyler 
Richard Wallace 
William Webster 



X 
X 



X 
X 



X 
X 

X 
X 



X 
X 



X 
X 
X 
X 



X 
X 



X 
X 



X 

X 

X 
X 



X 
X 
X 

X 
X 



X 
X 
X 
X 



X 
X 



X 
X 
X 



Pecent of Staff | 



I3Z 



20Z 



I6Z 



41Z 



43Z 
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BOSTON COLLEGE 



School Dists, 



Labs 



Cen ters 



UNIV. KANSAS 



City of Boston 


3 

Kansas Dists* 


3 

Indianapolis 


1 

Texas Consortium 


Schoo I Conun i t te e 






(Eight largest 


K. Greenwich, RI 


Jefferson County 


]^ 

Kalamazoo 


urban schools 


Hudson, NH 


Ken tucky 




dis tricts in 


Lynnfield, MA 




1 

Cincinnatti, OH^ 


Texas 


Middleborough MA 


1 

Waterloo, lA 




New Bedford, MA 




Saginaw, MI 


Fort Worth 


Newport, RI 








N.BrookfieId,MA 




Coms tock, MI 


Atlanta 


Paw tucket, RI 








Sanborn , Kings ton 




Hillsborough, 




Newton, RI 




County, FA 




Silver Lake, 






Kingston, MA 




De troit 




We have letters 




Lansing, MI 




from 31 school 






districts in 




Springfield, OR^ 




Massachusetts , 








New Hampshire, 




Shaker Hts., OH^ 




& Rhode Island 








which have com- 




Toledo"^ 




mitted them- 








selves to col- 








labor tion with 








the Center par- 








ticipating in 








the state mini- 








mum competency 








testing program 










MCREL^ 


NWREL^ 








Midwest Lab^ 








SWREL^ 








AEL^ 




LRDC^ 




2 

IRT^ 


UTTEC^ 


CSOS^ 




NCVTE^ 








Oregon Ctr.^ 








(CEPM) 
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BOSTON COLLEGE 



?rof. Society 



fov't. Progs. 



EAs 



ndlviduals 
and Others 



UNIV. KANSAS 



W.M.U. 



CAPE^ 

NCME^ 

NPTA^ 

APA^ 

ASCD^ 

AFT^ 



ACE' 



ERIC T/M" 
JDRP/NDN^ 



MassI 

South Carolina 



Ron Hambleton 



1 Warlharan 



Swamlna than 
Peter Alraslan 
Walter Haney 
Lary Ludlow^ 
George Madaus^ 



Kansas 



DALLAS SCHOOLS 



Joint Comra* 

Black Coll. 2 

EN^ 

NEA^ 

AASA^ 

ECS^ 

APGA^ 

NASSP^ 

NAESP^ 

NSBA^ 



Maryland 
Michigan^ 
Minnesota^ 
ECS^ 



LA.' 



Yvonne Lincoln" 
3 

Judith Hurry 

Susan 

Embrc tson 

Douglas 
Glasnapp 

4 

Egon Guba 



Mike Scriven'' 



Daniel L. 
Stuff I^.bs£ 



Egon Guba 



Gil Austin*" 



Mary Anne Bunda 
Arnold Gallegos^ 



AERA Div. A^ 



TEA 



Cordelia 1 , 
Alexander" 



Bill Denton" 



Maria Luiss 
Gonzalez 



Robert Grobe* 



Thomas Heiry 
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BOSTON COLLEGE 



Individuals 
and Others 



■Ion £d Groups 
2xtemal 



UNIV. KANSAS 



Joseph F Mlulla 
Gil Austin^ 



Ed Me yen 

David Miller"^ 
4 

Joha Poggio 



BOSTON COLLEGE 



UNIV. KANSAS 



Cci^sumers Union 



y.M.u. 


DALLAS SCHOOLS 


Thomas Ryan^ 


4 

Robert Mendro 


Carol Tittle^ 
Sri Kanta Rao^ 


Napoleon 
Mi tchell 


Bob Travers 


George Olson 


3 

Dick Jaeger 


Bl^l Webster 


Lyke Thompson^ 




Bill Cooley^ 




E. Diamond^ 




3 

John Sandberg 




4 

James Sanders 




3 

Robert Travers 




S. Mayo^ 




R. Tvler"^ 




W.M.U. 


DALLAS SCHOOLS 



I. 



Communlca tlon 
Coordination 
Coopera tion 
Collabora tor 
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Key Informnt Survey Scnaary 



Respondent 



Priorities And Recomendations 



National Urban League 
Ms* Stephanie Robiason 
Director of Education 



Center for Law and Education 
Ms. Sue Jackson 
Board Member 



Council for Basic 
Mr. Dennis Gray 
Deputy Director 



Education 



American Association of School 
Adminis tra tors 
Dr. Richard D. Miller 
Executive Director 

Mexican American Legal 
Defense Fund 
Mr. Ron Vera 
Attorney 6c Director 
Higher Education 
Project 

Southwest Educational 
Development Laboratory 
Dr. Preston Kronkosky 
Executive Director 



Validating new approaches and programs 
Documenting effective programs 

Disseminating information replicating effective 
programs 

Develop alternatives to traditional multiple 
choice tests that reinforce critical thinking, 
problem solving, and effective writing and 
speaking 

Develop and catalogue existing testing devices 
that are fair and equitable and relate to 
different learning styles and areas 
Set a series of goals and objectives for testing 
and its effects and work toward them 
Establish models of parent-student-staff evalua- 
tions and utilizations of evaluations 
Develop more effective ways of describing 
educational outcomes 

Educating lay audiences about proper uses of 
testing; including parents, policymakers , 
journalists, and other citizens 
Linking testing to curriculum and instruction 
Teaching teachers how to use testing properly 
Moving beyond paper and pencil testing to 
performance reviews 

Testing programs for "higher order" thinking 

Test validation 
Dissemination of results 



Procedures for verifying test scores on 
standardized tests 

Strengthening the capability of administrators 
to utilize and interpret standardized tests 
Validating the policy uses to which 
standardized tests are put 

Synthesize and disseminate the existing knowledge 
bases on educational sting 

Serve as a clearingho^ji v > for collecting, storing, 
retrieving, and disi?emiaa ting informar.ion on 
school testing evaluation and standards 
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Public Education 

Association 

Ms • Jeanne Frank I 

Executive Director 



University of North Dakota 

Center for Teaching and 

Learning 

Dr. Vito Perrone 

Dean 



Develop alternatives to the traditional multiple 
choice tests that better reinforce critical think- 
ing, problem solving, and effective writing and 
speaking 

Designing procedures for testing and evaluation to 
provide systematic and fair placement and promo- 
tion of students 

Setting appropriate standards guidelines against 
which student tesC performance may be compared 

Development of testa that? focus on thinking, 
assess developmental levels, are presciptive/- 
diagnostic, teacher useable and manageable, and 
measure growth in skill 

Focus on appropriate utilization of tests for 
promotion, on setting st&n<iards for teacher made 
tests(most closely allied to classroom work), and 
testing of professional, as well as, student 
competencies 

Long range study of the predictive value of test 
resul ts 

.Development of multi-racial, multi-cultural non- 
verbal test to serve all LEP children 
Study the equity implications of private coaching 
schools for tests like SAT 

Study of equity in relation to testing, testing 
and curriculum related to teaching and learning, 
alternative assessment mechanisms 



The James Russell Lowell 
School 

Dr. William D. Corbett 
Principal 



National Council of English 
Teachers 

Dr. Yetta Goodman 
President 



Michigan-Elementary and Middle 
Schools Principals Association 
Dr. William Mays, Jr. 
Executive Director 



Examine the effects on education in general and 
student writing in particular of the intense use 
of multiple choice, fill in testing 
Explore alternate methods of testing 
Establish oj^euness in testing at all levels 
Establish ERIC test dissemination contract 
Study the impact of test-directed placement policy 
on special populations 

Develop testing and evaluation techniques that 
facilitate a range of curriculum models 
Study of truth in testing and of legality in 
relation to tc-5ting 

Study the impact of testing on minority 
populations 

Serve as a clearinghouse for disseminating 
educational research 

Study of test and evaluation procedures for the 
fair placement and promotion of students 
Effects of linking school funding to test score 
and of linking promotion/ pay raises to test scores 
Study policy Mking process by which standards for 
assessing students are set 
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National Science 
Teachers Association 
Dr. Edward P. Ortleb 
President 



National Education Association 

Dr. Bernard McKenna 

Program Development Specialist 



Virginia-Department of 
Education 

Dr. Gerald W. Bracey 
Special Assistant for 
Policy 



Kansas -National Education 

Association 

Dr. Marilyn Flannigan 

Dire tor of Instructional 

Advocacy 

Association for Supervision 
and Curriculum Development 
Dr. Gordon Cawelti 



Develop alternatives to the traditional multiple 
choice tests that better reinforce critical think- 
ing, problem solving, and effective writing and 
speaking 

Develop evaluation techniques applicable zo local 
instructional programs 

Link testing more closely to inrtruction 
Study positive and negative effects of developing 
educational policy on tests result 
Develop methods to aid teachers and school 
administrators to interpret test data 
Develop appropriate guidelines for setting 
standards against which student test performance 
may be compared 

Study the policy oiaking process by which standards 
for assessing students are set 

Evaluation of student learning in the broad sense 
DevelopTient of multiple criteria for evaluating 
student progress 

Develop assessment for diagnosis, prescribing 
remediation, and planning instruction in general 

Development of proficiency based assessments 
Study the appropriate role of assessments in 
policy 

Study the current and desired relation of 
instruction to assessment 
Develop long-term strategies for data basf 
Study of norm-referenced testing in relatiow t<i. 
their construction 

Develop more effective ways of describit;, 
educa tional outcomes 

Develop tes.:ing practices which asoist V'Afj.^XbihliV. 
student populations 



Study discrepencies of national norm«refe. _ced 
tests and local curriculum 

Help districts develop subject o^Atter lists CrXt 
their curriculum as an alternative, to re?,ianr.e and 
preoccupation with SAT or NRT as indfcatcrs of 
excellence 

Study the effects of testing not specifically 
linked to content, such as "gate testing", which 
is responsible for promotion, advancement, 
placement* or acceptance 

Develop methods fo\: presenting test resultsCnorm 

crit^i^^lon referenced) to parents and other lay 
AUdienc^:^ in meaningful and us<?,ful manners 
Develop valid and reliable standards for passicg, 
or "cut points" 
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NIE Planning Survey: Responses to Open-Ended Questions 



Key 

Dimension A 
Code 

53 
6 
0 
2 
4 
0 
1 
2 



1: Testing 

2: Evaluation 

3: Standards 

4: Testing & Evaluation 

5: Testing & Standards 

6: Evaluation & Stndrds 

7: All Three 

8: None of the Above 



Dimension B 
# Code 
10 
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Code 
A B 



5 
8 
2i 
4 

13 



_2 

68 



1: Enhance curriculum & instruction 
2: Promote diagnostic/prescriptive uses 
3: Di sseminate information to constituents 
4: Enhance equity/reduce inequity 
5: Enhance planning/policy uses 
6: Reduce problems 'wi th traditional 

approaches/promote uses of alternative 
approaches 
7; Promote student growth/higher order 
learning 



Conent 



Section A: MISSION GOALS/OBJECTIVES FOR THE R&D CENTER 

3 Develop public information programs on limitations of standardized 
tests . 

3 Study & disseminate information on the impact of minimal competency 
testing programs on minority education. 

4 Development of multi-racial, multi-cultural nonverbal test to serve 
all LE? children. 

4 Establish the practice of test openness in all publically supported or 
requ^ red tes ting « 



4 
4 
6 
6 
7 
4 



See Golden Rule settlement (enclosed openness in testing article?). 
Investigate testing abuse. 

Long range study of predictive value of test results. 
Develop in process methods of evaluations. 

Multiple criteria for evaluating student learning progress. 
Develop standards for truth in testing. 
Section B: INTBCRATION OF TESTING WITH STUDENT LEARNING 
1 1 How to keep testing from dominating curriculum. 

1 4 Examination of the discriminatory features of tests for specific 
populations including boys for reading in early grades, women, 
multiligual populations, etc. 
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Code 

A B CoBsent 



8 4 Issues related to cultural bias. 
Section C: ETHICAL AND POLITICAL ISSUES 

1 4 Equity implications of private coaching schools for tests like SAT 
(see item C4, the effects and implications of tes twiseness) • 

1 4 When tests are used to judge a person, that person should have the 
absolute right to view the test and his/her scored answers post 
adminis tra tive ly • 

1 4 Sould tests be used for placement and promotion? Can any one test be 
a gatekeeper? 

1 4 The use of tests as the major means of discrimination in our society. 

1 4 The use of tests to screen out cthrewise qualified people from 
programs, jobs, education, professions, etc. 

Section D: TEST INFORMATIOH UTILIZATION 

1 6 Informal observational techniques in testing (Kidiva & Ching) . 

1 6 Limitation of test- teach- test paradigm. 

2 1 Evaluation (observational techniques) as a continuous part of 

curriculum development and instruction. 

Section £: STANDARDS- SETTING 

1 2 Testing assessment for diagnosis. 

1 2 Testing assessment for prescribing remediation. 

5 4 What are the purposes for sett:^ng standards? Everything involving 
standards encorporates a view of test as "gatekeeper." This issue 
itself needs to be examined in terms of its purposes relevant to 
society and various groups within the society. 

Section G: MOST CRITICAL NEEDS 

1 1 ;.inking testing to curriculum and ins true tion« 
1 1 Testing & curriculum/ teaching & learning. 

1 1 Major problem is incongruity of national norm referenced tests and and 
local curriculum--need more work similar to IRT pointing out 
descrepancies. 

1 1 Helping instructors develop subject matter tests on their curriculum 
vs. prescribing with SAT or NRT as indicators of excellence. 

1 1 Assessment for planning instruction generally. 
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Code 

A B Cosaaent 



1 1 Relation of instraction to assessment (current & desired). 
1 2 Prescriptive/diagr.ostic tests. 
1 2 Assessment for diagnosis. 
1 2 Assessment for remediation. 

1 3 Educating Uy audiences about proper uses of testing: parents, policy- 
makers, journalists, other citizens 

1 3 Teaching teachers how to use testing properly- 

1 3 Dissemination of results. 

1 3 Al. Synthesize & disseminate existing knowledge bases on educational 
tasting. 

1 3 Description of how NRT'^s are really constructed. 
1 4 Equity in relation to testing. 

1 4 Establishing openness in testing at all levels. 

1 4 Truth in testing. 

1 4 Legality and testing. 

1 5 Appropriate role of assessment in policy • 

1 6 Moving beyond paper and pencil testing to performance reviews. 
1 6 Test validation. 

1 6 A8. Develop alternatives to the traditional multiple choice tests that 
be tter reinforce critical thinking, problem solving, and effective 
writing and speaking. 

1 6 Teacher useful and manageable tests. 

1 6 Alternative assessment mechanisms. 

1 6 Misuse of standardized entrance tests by coUeges and universities 
throughtout the country. (See included document.) 

1 6 Examining the effects on education in general and student writing in 
particular of the intense use of multiple choice, fill in testing. 

1 6 Exploring alternative methods of testing, 

1 6 Need for proficiency-based assessment. 

1 7 Testing programs ror "higher order" thinking. 

ERIC 



Code 

A B Coment 



1 7 Tests that focus on thinking competencies. 
1 7 Tests assessing developmental levels. 

1 7 Tests that measure growth in skill. 

2 4 Impact of evaluation on minority populations. 

2 7 Evaluation of student learning in the broad sense. 
2 7 Multiple criteria for evaluating student progress. 

4 1 Testing and other evaluation techniques that facilitate a range of 
curriculum models. 

4 5 C3. Testing and evaluation procedures for systematic and fair 

placement and promotion of students. 

5 5 El. Appropriate guidelines for setting standards against which student 

test performance may be compared. 

7 3 A2. Serve as a clearing house for collecting, storing, retrieving, and 

disseminating information on school testing, evaluation, and 
standards. 

8 5 Need for strategic thinking about data (as opposed to quick fix 

remedies based on short term information). 

Section H: ADDITIOHAL RECOHMENDATIOHS 

1 4 Focus on appropriate utilization of test results for promotion, etc. 

14 1 assume they will focus on testing of professional competencies as 
well as student competencies. 

14 1 admire your effort to establish a national testing center. It is 

long overdue. The center should be staffed with a wide variety of 

professionals including knowledgable practitioners at all levels of 
the educational spectrum. 

5 1 Focus on how to set standards for teacher-made tests-^most closely 
allied to classroom wo\k.. 
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Reviewer's Comments-Planning Grant Competition 
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NIE PLAMHING GRANTS COMPETITIOH SIWMARY REPORT OF REVIEWERS' COMHEHTS 

Sumary of Reviewrs' Judgaents of Proposal Coaponents in the Forta of 
<ReTlewir> judges the <coiponent> as <judgBent>. 



<Re viewer) judges 



the <coniponent> 



as <judginent>. 



Introduction 

1) The panel 



application 



Major Strengths 

1) A majority of the proposal on each of the evaluation 
panelists criteria 

2) Each of the panelists summary of the history of developments 

in testing & evaluation 

3) Each of the panelists analysis of the in:errelatednes3 of 

testing, evaluation, & standards 



4) 



5) 



7) — 



developing a conceptual framework 
related to the Center mission 

plan of operation 



organizational plan in general 
strategies for outreach in particular 



8) Several panel Dierabers applicant's concern & plans for addres- 

sing the issuts ci the underserved 



9) 

10) — 

11) The panel 



overall quality of proposed staff 

(staff) in psychometric aspects of 
testing 

(staff) Stufflebeam & Medaus 
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mean rank of 2.4. 



generally strong. 



strength, 



strength. 



good job. 



^•ell structured, appropriate for meetln", 
stated objectives of planning process, 

exceptionally strong. 

exceptionally strong. 

strength. 

major strength, 
somewhat lacking. 



noted accomplishments of. 
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as <judginent>. 



12) — 

13) — 

14) — 

15) — 

Major tfeaknesses 

1) Each panel member 

2) Panel members 

3) — 
0- 

5) 

6) The panel 
7) 

8) l.ii panel 



(staff) proposed consultants 
(staff) the group 

evidence of prior collaboration among 
Institutions 

"track record" of prior Institutional 
accomplishments 

mission statement 

important issues in the field & how 
these issues might be addressed 

psychometric and and other technical 
issues regarding testing 

proposal 

scope of the plan of operation 
extensive schedule of interviewing 

design 

inclusion of teacher evaluation as a 
focus of the mission statement 



"breadth of experience." 

"good mix of scholars L practitioners." 

strength. 



strength. 



the primary concern- overall vagueness 
and lack of specificity. 

lacking sufficient specificity. 



far too little attention. 



somewhat imba lanced toward evaluation to 
the neglect of important testing 
concerns. 

much too extensive, 

might lead to difficulty in consolidat- 
ing the obtained input. 

seems to delay discussion of r'^search 
priorities . . . thus compresses the 
hard choices into a "painfully narrow 
time span," 

questioned its apprcpriataness for a 
ce.Lcr on "student" testing, evaluation 
& standards. 



er!c4 



<Reviewer> judges the <coinponent> 



as < judgment). 



Somry of Analyels 

1) The panel majority proposal 

2) Only one of the panel proposal 
members 



3) — 

4) — 

5) - 

6) — 
7) 



8) — 



plan of operation 

strategies for outreach 

what can be accomplished within a 
particularly short time fraine 

discussion of critical technical and 
psychometric issues 

mission statement 



operating plan as a vehicle for 
determining the Center's agenda 



a strong application that is fundable, 
"fundable with reservations." 

particularly strong, 
particularly strong, 
perhaps overly ambitious. 

somewhat hcking. 



primary criticism-Vfgue and overly 
general giving little basis for judging 
the direction the Center effort might 
take. 



the strength that overcomes the above 
weaki;':ss. 
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Content Analysis of Request for Proposal 
for NIE Center on Student Testing, 
Evaluation, and Standards Setting 
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SUtmgl OF 1985 RFP FOR Kllfi CENTER ON STUDENT TESTING, EVAUMTION, Alffl) STANDARDS 
IJOT OF (MZ THOSE STATEMENTS DIRECTLY RELATED TO TESTING 

TOPIC 

T E S where T « Tt^STING, E « EVALUATKM, S = STANDARDS. 



Definitions 

Testing: ?onnal means to assess student educational outcomes. This may refer 
to paper L pencil testing, performance testing, and other forms of testing. 

Ajsseosaient: Not explicitly defined. Usually implies "testing," other times 
implic?.s "testing-based evaluation." 



Introduction 

Context 



1. Testing & evaluation activities play a central role in American schools. 



X X 


2. 


Pare ."Its, educators, & policymakers at state, national, & local levels ask 
questions about a) how their children plus specific schools & schools in 
general are doing, & b) how can they be improved. 


X X 




Testing & evaluation activities can answer many of these questions. 


X X 


5. 


Policymakers & legislators often act upon testing & evaluation results. 


XXX 


6. 


The perception of crisis in current educational standards is largely based upon 
results from testing & evaluation. 




Strengths/Opportunities 


X 


i. 


Test developers & publishers have produced many standardized tests for large 
scale use in American education, e.g., norm-referenced tests, the SAT, & other 
test of aptitude & capacity for post secondary education. 


X X 


2. 


The National Assessment of Educational Progress has provided periodic 
assessments of the nation's overall progress in education for the past five 
years. 


X X 


3. 


State assessments of different kinds have provided much information for 
educational decision making at state & local levels. 


X X 


4. 


Progress, is being made in the area of local testing & evaluation because of a) 
advances in the broader arena— state & national, & b) developments in learrdng 
theory & technology, testing & evaluation. 



ERIC 



rs9 



TOPIC 

T E S where T « TESTING, E = EVALOmCW, S = STANDARDS. 



Needs/Concems/ProbleHB 

XX !• The need for better testing & evaluation continues. 

XX 4. A need for better testing & evaluation tools already exist, locally. 

XX 5. The benefits attained in developing testing & evaluation methodologies for 
state & national use have not always been transferable to the local level. 

XX 6. Concerns at the local level frequently focus on the fairness & utility of 
resting & evaluation methods for dealing with individual students, classrooms, 
& other groupings. 



XX 7. 



A need for more readily available & s>stematic information based on testing & 
evaluation to help diagnose, place, instruct, & promote individual students 
exists. 



XX 8. A need exists for tools to assess the merits of individual classroom & school 
programs more satisfactorily because statewide or norm-referenced test results 
are often the only ones available with which to evaluate them. Such tests are 
not good for assessing the effect;iveness of discrete local instructional 
activities. This need will grow as school districts are called upon to 
inclement con?)rehensive reform legislation initiated by many states. 



XX 9. 



A need still exists to identify ways to make testing & evaluation information 
more useful to its intended audiences. 



X 10. Part of the challenge is technical, e.g., to present psychometric information 

clearly & succinctly or to package & distribute information in timely & 
effic: nt ways. 

XX 11. Part of the challenge entails better translation of teachers' & other users' 
language to that of testing & evaluation & visa versa. 

X X 12. It also means integrating the perspectives of teachers & oth-ri into testing & 
evaluation so that the different con^onents of school work more harmoniously 
together. 

XX 13. The amount of attention & resources placed upon local testing . evaluation has 
historically lagged behind that accorded to the issues beyond the local lev^l. 



Scope of Center Mission 
Specifications: Mlssion/ShoiildsAfays 

XX 1. The primary mission of this Center should \>e to increase the contribution that 
testing & evaluation can make to local school improvement. 

X X 3. The Center ought to make demonstrable co' tributions toward meeting two goals: 
1) a rise in test scores, and 2) a perc^cion that educational standards have 
in?)roved. ExanqpJes of such contributions include: 
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XX * developing more effective, efficient, & fair methods of testing & evaluation 

in schools; 

XX * demonstrating ways to make testing & evaluation practices more equitable to 

the individual students; & 
XX * demonstrating ways to make these practices more useful co all concerned at 

the local level. 

XX 4, The Center should be cognizant of the impacts of its research upon policy at: 
the state & national level & maintain contacts with officials at these levels > 
including state research & evaluation directors & state assessment directors^ 

XX 6. Testing & evaluation in the core subjects of reading, language arts, writing, & 
mathematics should be en5)hasi2ed. The mix of these subjects is optional. 

XX 7. Research on testing & evaluation in other subjects such as social studies, the 
arts & humanities, or nonachievement outcomes like attendance or dropout rates 
may be addressed if it can be related to other work on the core subjects, or 
exceptional expertise or unusual opportunity presents itself. 

XX 8. The Center should address testing & evaluation issues in elementary & secondary 
education. 

X 9. Issues at the secondary level may include the use of proficiency, minimum 

con5)etency, or other achievement tests used to regulate high school graduation. 



Strengths 

XXX 1. Good testing & evaluation can provide information with sufficient precision, 
timeliness, & utility to assist in the debate & decisionmaking about standerds. 

Research That May be Performed 
General Specifications: Shcyilds/Hays 

XX 2. Basic research should be framed so that its potential value to testing & 
evaluation in schools can clearly be seen. 

X 3. Research may embody theoretical perspectives of classical test design, item 

response theory, Bayesian statistics, cognitive science & psychology, subject 
matter domains & technology^ 



X X 



Research may involve practical perspectives of testing & evaluation procedures 
& use. 



XX 5. Research may involve both laboratory & field work, but a substantial proportion 
of the research should be conducted on the school siter & involve the active 
participation of sch^l personnel such that a reasonable observer ought to be 
able to conclude that the extent of research conducted in schools is 
commensurate with the Center's mission to help school improvement through 
testing & evaluation. 
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Technological /l^ipllcatlons 



1. Diagnostic Testing 

X Needs: a) A need for efficient & perceptive forms of diagnostic testing for 

use with Individuals or smaller groups of students exists. 
X Strengths: a) Progress In diagnostic testing has been aided by advances in 

mlcrocon?>uter technology, cognitive science, & psychometric theory, 
X Questions: a) How can specifications for such tests be best written, 

particularly regarding content? 
X b) How short can such tests be and still be reliable and valid measures of 

student learning? 

2. Tailored or Adaptive Testing 

X Strengths: a) Such tests offer the possibility of reducing testing time, 

thereby making classroom testing more efficient, 
X b) Such testing adapts the starting point and questions to be asked to the 

ability level of the individual student. 
X c) A considerable body of theoretical knowledge of such tests exists. 

X X Needs: a) Further research to establish the validity as well as the efficiency 

and utility of such testing might be conducted. 

3. Nonintnislve Testing, in which instructional software is designed to ascertain 
what the student is learning by monitoring the level and sequence of materials 
attempted. 

X Strengths: a) The student is tested without taking a "test." 

b) Capacities in technological hardware and software are Increasing. 
X c) Learning and psychometric theory are advancing. 

X X Needs: a) Better testing & evaluation in schools is needed. 



Locally Responsive Tests 

1. Tests That Measure the Specific Objectives of Local Instructional Prograw 
Needs/Problems: a) An increasing need for such tests exists. 

b) A good measure of a local reading program at the intermediate grades may be 
needed. 

c) A conventional norm-referenced test may be neither matched with the content 
of the local program nor conclusive as to what knowledge the students actually 
learned. 

d) Large numbers of subject domains have to be assessed, making the local cost 
of developing tests for all of them very high and the workload excessive. 
Strengths: a) The development of criterion-referenced tests, which assess 
mastery of specific subject domains, has helped meet school needs in this area. 

2. Itea Banks 

Strengths: a) item response theory has increased the capacity to generate 
banks of test items that may be used interchangeably by school districts and 
state education agencies to create customized test? for their own use. 

b) Item response theory makes possible the definition ot the statistical 
properties of individual test items without reference to the rest of the test 
with which they were developed. 

c) Alternatives to item banks may also meet the need for locally responsive 
tests . 
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X (^lestions: a) How can test specifications best be developed for local use? 

X b) Under what conditions is item banking a productive test development strategy 

for local use and how can those conditions be maximized? 
X c) When are other methods more appropriate? 

X Needs: a) Schools and districts need locally responsive tests that are not now 

available. 



Curri culuH-Tes t Match 

Needs: a) Research might further clarify the consequences of the relationship 
between the curriculum in school districts and the tests used in those 
districts. 

Questions: a) When tests indicate that students have not mastered a subject, 
to what degree are the tests an accurate measure of what was taught? 
X Strengths: a) Further research in this area & the others mentioned above would 
lead to more conclusive measures of how well students are doing and what 
present educational standards are. 



Helping with Sducatlonal Standards 

XXX Needs/Problems: a) Current efforts to raise educational standards involve many 

aspects of testing & evaluation whose methodology is imperfect or undeveloped, 
e.go, the use of proficiency tests, etc., to regulate promotion and graduation 
at the local level requires instruments that are fair to all students, 
particularly lower-achieving ones and those near the threshold which has been 
established as the "passing" pr "cut" point. 

X X Questions: a) How can subject matter domains be better specified and tests 

made more discriminating at the critical "cut points"? 

XX b) What local nonpsychometric procedures (e.g., methods for standard setting) 

can be developed to make pass-fail decisions more accurate and equitable? 



Assisting Tea che rs and Parents with Testing and Evaluation 

X X needs /Problew: a) Many practical methodological concerns exist at the local 

level about the capacity of teachers, other staff, & parents to use testing & 

evaluation for day-to-day purposes. 
XX b) Teachers & parents need means to help them understand effective testing & 

evaluation practices and adapt such practices to the needs of classrooms. 
X X Questions: a) How can testing & evaluation results best be made responsive to 

the needs of teachers and other local staff members? 
X b) What degree of translation between goals of teachers and those of test 

developers is necessary to turn teachers' instructional objectives into useful 

tests and test items? 

XX c) What are the best strategies to engage teachers, other local staff members, 

and i^arents in testing & evaluation methods? 
XX d) Wnat are the limits to such strategies? 
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Information Systems 
Strengths/Context : 

b) Such systems may contain results 
data, such as statistics on class 
attendance, and dropouts. 



of testing programs, and other evaluative 
grades, homework, disciplinary measures. 



Evaluation and Testing Uses: How to Gain More Use of Credible Informtion 

XX Problems/Context: a) People in all quarters of education display varying 

degrees of reluctance to use good testing & evaluation. 
XX c) In some cases, people have been "burned" by the use of testing & evaluation 

information that actually turned out to be faulty in one or more aspects. 
XX d) The research topics suggested above address the 'problem of faulty or 

teclinlcally deficient information through technical or applied perspectives 

while this topic en?)hasi2es better understanding of how to gain more use of 

credible testing & evaluation information. 
X X Questions: a) What barriers exidt to the use of credible testing & evaluation 

information? 

XX b) How can the discovered barriers be overcome? 

XX c) How can the production of good Information be made less costly? 

X X Shoulds: a) Research in this area should help establish what the potential of 

the information age" at the local school level actually is. 
XX b) Research should not address general topics such as what distinguishes the 

school as an information-seeking institution. 
XX c) Any studies of Information use should be tightly tied -.o discrete Issues of 

testing and evaluation practice. 



X X 



Organization and Staff inj 

Specifications: Shoulds 

5. Staff members with practical perspectives on testing & evaluation procedures 
based on experience in schools are expllc^ Qy desired as part of the Center's 
overall staffing. 



X X 



Di88«^natlon 

Specifications: Shonlds/ISLlls 

2. The Center will mkr its contribution i:o scholarly journals in testing & 
evaluation. 



X X X 4. It will serve as a resource to those in schools & elsewhere concerned with 
improving schools in the areas of testing, evaluation, and standards, e.g., 
school district directors of research & evaluation, directors of testing, 
teachers & teacher organizations, state research & evaluation directors, and 
state assessment directors. 



X 




5. The Center's work will be made widely available to educational test publishers 
and 8iiil].ar organizations. 

6 
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XX 8. Cooperation with the ERIC Clearinghouse on Tests, Measurement, and Evaluation 
is expected. oj-uauxun 



Collaboration with Other Centers 
Context/Needs/Strengths 

X X X 3. The NIE Center on Testing, Evaluation and Stand?.rds will have special skills 
that may be selectively used on challenges of the other Centers, e.g., it might 
allocate some staff time & resources to work collaboratively with another 
Center on a problem of mutual interest. Ideally, this would happen where both 
Significant testing & evaluation issues and significant substantive issues were 
raised in a particular research problem. Other situations can also be imagined. 

Examples 

XX 1. Writing (writing assessment in the field) 

XX 2. Reading (reading assessment in the field) not part of this grants competition 

X 6. Postsecondary Teaching and Learning (basic skills or other testing used for 

admission or placement of students in postsecondary institutions) 

XX 7. Teacher Quality and Effectiveness (methods for assessing teacher competencies) 
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T E S where T = lESTING, E = EVALGATKW, S =« STANDARDS. 

Deflnltlcms 

X Evaluation: Process used to place student educational outcomes in their 

specific context, -relate them to instructional and other factors, weigh their 
significance, and make decisions based on them*, 

X X Assessment: Not explicitly defined. Usually in?)lies "testing," other times 

implies "testing-based evaluation." 

Introduction 

Context 

XX 1. Testing & evaluation activities play a central role in American schools. 

XX 2. Parents, educators, & policymakers at state, national, & local levels ask 
questions about a) how their children plus specific schools & schools in 
general are doing, & b) how can they be in5)roved. 



XX 3, 
X 4* 
XX 5. 



Testing & evaluation activities can answer many of these questions. 

School personnel are held accountable for their results. 

Policymakers & legislators often act upon testing & evaluation results. 



XXX 6. The perception of crisis in current educational standards is largely based upon 
results from testing & evaluation. 



Strengths/Opportunltli 



.es 



XX 2. The National Assessment of Educational Progress has provided periodic 
assessments of the nation's overall progress in education for the past five 
years. 

XX 3. State assessments of different kinds have provided much information for 
educational decision making at state & local levels. 

XX 4. Progress is being made in the area of local testing & evaluation because of a) 
advances in the broader arena— state & national, & b) developments in learning 
theory & technology, testing & evaluation. 
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XX 1. The need for better testing & evaluation continues. 
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X X 



X X 



XX 4. A need for better tp.sting & evaluation tools already exists locally. 

XX 5. The benefits attained in developing testing & evaluation methodologies for 
state & national use have not always b«en transferable to the local level. 

XX 6. Concerns at the local level frequently focus on the fairness & utility of 
testing & evaluation methods for dealing with individual students, classrooms. 
& other groupings. 



XX 7. 



A leed for more readily available & systematic information based on testing & 
evaluation to help diagnose, place, instruct, & promote individual students 
exists. 



XX < . A need exists for tools to assess the merits of individual classroom & school 
programs more satisfactorily because statewide or norm-referenced test results 
are often the only ones available with which to evaluate them. Such tests are 
not good for assessing the effectiveness of discrete local instructional 
activitieF. This need will grow as school districts are called upon to 
implement cc^rehensive reform legislation initiated by many states. 

XX 9. A need still .-ists to identify ways to make testing & evaluation information 
more useful to i :r. intended audiences. 

XX 11. Part of the challenge entails better translation of teachers' & other users' 
language to that of testing & evaluation & visa versa. 

12. It also means integrating the perspectives of teachers & others into testing & 
evaluation so that the different components of school work more harmoniously 
together. ■' 



13. The amount of attention & resources placed upon local testing & evaluation has 
historically lagged behind that accorded to the issues beyond the local level. 



X X 



X X 
X X 
X X 



Scope of Center Mission 
Specifications: Mssion/Stxnilds/Mays 

1. The primary mission of this Center should be to increase the contribution that 
testing & evaluation can make to local school in?>rovement . 

3. The Center ought to make demonstrable contributions toward meeting two goals- 
1) a rise in test scores, and 2) a perception that educational standanis have 
ln?>roved. Examples of such contributions include: 

* developing more effective, efficient, & fair methods of testing & evaluation 
in schools; 

* demonstrating ways to make testing & evaluation practices more equitable to 
the individual students; & 

* demonstrating ways to make these practices more useful to all concerned at 
the local level. 
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4. The Center should be cognizant of the impacts of its research upon policy at 
the state & national level & maintain contacts with officials at these levels, 
including state research & evaluation directors & state assessment directors. 

6. Testing & evaluation in the core subjects of reading, language arts, writing, & 
mathematics should be emphasized. The mix of these subjects is optional. 

7. Research on testing & evaluation in other subjects such as social studies, the 
arts & humanities, or nonachievement outcomes like attendance or dropout rates 
may be addressed if it can be related to other work on the core subjects, or 
exceptional expertise or unusual opportunity presents itself. 

8. The Center should address testing & evaluation issues in elementary & secondary 
education. 



Strengths 

XXX 1. Good testing & evaluation can provide information with sufficient precision, 
timeliness, & utility to as^sisC in the debate & decisionmaking about standards. 



Fesearch That May be Performed 
General Specifications: Shoulds/Mays 

2. Basic research should be framed so that its potential value to testing & 
evaluation in schools can clearly be seen. 

4. Research may involve practical perspectives of testing & evaluation procedures 
& use. 

5. Research may involve both laboratory & field work, but a substantial proportion 
of the research should be conducted on the school sites & involve the active 
participation of school personnel such that a reasonable observer ought to be 
able to conclude that the extent of research conducted in schools is 
commensurate with the Center's mission to help school improvement through 
testing & evaluation. 



Technological Applications 

1. Diagnostic Testing 
Questions: 

X c) What kinds of diagnostic information have practical value for the classroom 

teacher and other school staff? 

2. Tailored oz Captive Testing 

X X Ifeeds: a) Further research to establish the validity as well as the efficiency 

and utility of such testing might be conducted. 
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. Hdnlntrusive Testing, in which instructional software is designed to ascertain 
what the student is learning by monitoring the level and sequence of materials 
atteo^ted. 

Needs: a) Better testing & evaluation in schools is needed. 



Helping with Educational Standards 

XXX Needs/Problems: a) Current efforts to raise educational standards involve many 
aspects of testing & evaluation whose methodology is imperfect or undeveloped, 
e.g., the use of proficiency tests, etc., to regulate promotion and graduation 
at the local level requires instruments that are fair to all students, 
particularly lower-achieving ones and those near the threshold which has been 
established as the "passing" or "cut" point. 



X X 
X X 
X X 



Assisting Teachers and Parents with Testing and Evaluatlra 

X X Needs/Problew: a) Many practical methodological concerns exist at the local 

level about the capacity of teachers, other staff, & parents to use testing & 
evaluation for day-to-day purposes. 

b) Teachers & parents need means to help them understand effective testing & 
evaluation practices and adapt such practices to the needs of classrooms. 
Questions: a) How can testing & evaluation results best be made responsive to 
the needs of teachers and other local staff members? 

c) What are the best strategies to engage teachers, other local staff members, 
and parents in testing & evaluation methods? 

XX d) What are the limits to such strategies? 



X X 



X X 

X x 

X X 




Information Systems 
Strengths/Context : 

b) Such systems may contain results of testing programs, and other evaluative 
data, such as statistics on class grades, homework, disciplinary measures, 
attendance, and dropouts. 



Evaluatloia and Testing Does: How to Gain More Use of Credible Infor«tlon 

X X Problems/Context: a) People in all quarters of education display varying 

degrees of reluctance to use good testing & evaluation. 

b) In part, a suspicion of statistics or computers exists, no matter how good 
the information they provide. 

c) In some cases, people have been "burned" by the use of testing & evaluation 
information that actually turned out to be faulty in one or more aspects. 

d) The research topics suggested above address the problem of faulty or 
technically deficient information through technical or applied perspectives 
while this topic emphasizes better understanding of how to gain more use of 
credible testing & evaluation information. 

Questions: a) What barriers exist to the use of credible testing & evaluation 
information? 

XX b) How can the discovered barriers be overcome? 

XX c) How can the production of good information be made less costly? 
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Shoalds: a) Research in this area should help estn^Vlsh what the potential of 
the "information age" at the local school level ^.v^^tv;;. ; ^ ,^ 

b) Research should not address general topics r what distinguishes the 
school as an information-seeking institution. 

c) Any studies of information use should be t.f^hti^ to discrete issues of 
testing and evaluation practice. 



Organization and Staffing 
Specifications s Shoulds/Kays 

5. Staff members with practical perspectives on testirig ^ evaluation procedures 
based on experience in schools are explicitly desired part of the Center's 
overall staffing. 



X X 
X X 
X X 



Dissendnation 

Specifications: ShouldsAttUs 

XX 2. The Center will make its contribution to scholarly journals in testing & 
evaluation. 

XXX 4. It will serve as a resource to those in schools & elsewhere concerned with 
improving schools in the areas of testing, evaluation, and standards, e.g., 
school district directors of research & evaluation, directors of testing, 
teachers & teacher organizations, state research & evaluation directors, and 
state assessment directors. 

XX 8. Cooperation with the ERIC Clearinghouse on Tests, Measurement, and Evaluation 
is expected. 

Collaboration with Other Centers 
Context/Ileeds/Strengths 

XXX 3. The NIE Center on Testing, Evaluation and Standards will have special skills 
that may be selectively used on challenges of the other Centers, e.g., it might 
allocate some staff time & resources to work collaboratively with another 
Center on a problem of mutual interest. Ideally, this would happen where boch 
significant testing & evaluation issues and significant substantive issues ware 
raised in a particular research problem. Other situations can also be imagined. 

Exanples 

XX 1. Writing (writing assessment in the field) 

XX 2. Reading (reading assessment in the field) not part of this grants competition 
X 4. Effective Elementary Schools (methods for evaluating effective schools) 
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Effective Secondary Schools (methods for evaluating effective schools) 
Teacher Quality and Effectiveness (methods for assessing teacher competencies) 
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Definitions 

Standards: People's values and judgments about what desirable school outcomes 
and processes are. 



Introduction 

Context 



X X X 6. The perception of crisis in current educational standards is largely based upon 
results from testing & evaluation. 



Needs/Concems/Probl 

X 2. Some o' the need is a consequence of the pablic scrutiny of educational 
standards. 

X 3. Although much of this emphasis is a result of federal & state initiatives, 
issues involving standards assume practical significance only in local efforts 
to improve the schools. 



Scope of Center Mission 
Specifications : Mlssioa/Shoulds/M^ 

3. The Center ought to make demonstrable contributions toward meeting two goals: 
1) a rise in test scores, and 2) a perception that educational standards have 
improved. 



Strengths 



XXX 1. Good testing & evaluation can provide information with sufficient precision 
timeliness, & utility to assist in the debate & decisionmaking about standards. 



Research That Hay be Perf orwd 

Currlculuar-Test Match 

X Strengths: a) Further research in this area & the others mentioned above would 
lead to more conclusive measures of how well students are doing and what 
present educatiot^l standards are. 
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Helping with Educational Standards 

XXX Needs/Problems i a) Current efforts to raise educational standards involve many 
aspects of testing & evaluation whose methodology is imperfect or undeveloped, 
e.g., the use of proficiency tests, etc., to regulate promotion and graduation 
at the local levi:l requires instruments that are fair to all students, 
particularly lower -achieving ones and those near the threshold which has been 
established as the "passing" or "cut" point. 

X X Questions: a) How can subject matter domains be better specified and tests 

made more discriminating at the critical "cut points"? 

XX b) What local nonpsychometric procedures (e.g., methods for standard setting) 

can be developed to make pass-fail decisions more accurate and equitable? 

Df.ssealnatlon 

Specifications: Shoulds/Wills 

XXX 4. It will serve as a resource to those in schools & elsewhere concerned with 
improving schools in the areas of testing, evaluation, and standards, e.g., 
school district directors of research & evaluation, directors of testing, 
teachers & teacher organizations, state research & evaluation directors, and 
state assessment directors. 

X 7. The Center's work on standards needs to b^ a resource to many educational and 
policy groups concerned with those issues at the local level. 

Collaboratirai with Other Centers 
GoQtezt /Needs/Strengths 

XXX 3. The NIE Center on Testing, Evaluation and Standards will have special skills 
that may be selectively used on challenges of the other Centers, e.g., it might 
allocate some staff time & resources to work collaboratively with another 
Center on a problem of mutual interest. Ideally, this would happen where both 
significant testing & evaluation issues and significant substantive issues were 
raised in a particular research problem. Other situations can also be imagined. 
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Definitions 

Local: Individual school districts, school buildings, & classrooms. 



Scope of Center Mission 
Specifications: Mlsslon/ShouldsyM^ 

2. Its resources ought to directly benefit educators and parents at the local 
level . 

5. The priniary educational interest to the Center's mission should be student 
educational outcomes related to achievement. 



Research That May be Perforaed 
General Specifications: Shoulds/kays 

1. The Center may conduct a mixture 6f basic & applied research such that the 
perspectives & findings of the basic & applied research will feed each ether. 

6. The research may involve networking of interested schools & school districts if 
such networking will advance the conduct or dissemination of the work. The 
exact mix of these components & perspectives is up to the Center. 

7. In general, the Center's research should be planned to have demonstrable 
utility within the project period of this grant. 



Infomatioa Systi 

Strengths/Context: a) A small number of school districts is using the 
potential of technology to make organizations more productive & efficient by 
pursuing the development of comprehensive management or instructional 
information systems. 

c) Different systems have been tailored for the use of teachers & other 
building personnel or for the use of school district policymakers. 
Needs/ProbleiB: a) The technology for efficiently gathering such information 
and making it accessible to all levels of a school district is not sufficiently 
developed or understood. 
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Questions: a) What is the best mixture of hardware and software for such 
systems? 

b) How can existing technology be blended and adapted for local uses? 

c) What content is most useful when included in such systems? 

d) What training and other activities need to be addressed to make them 
workable? 

Evaluation and Testing Uses: How to Gain Jfcre Use of Credible Inforwation 
Probless/Context : 

b) In part, a suspicion of statistics or computers exists, no matter how good 
the information they provide. 



Other Research 

Specifications (Shc^rds/Hays): a) The foregoing lines of research do not 
exhaust possibla lines of research that may be co/ered. 

b) All of the above line of research need not be performed. 

c) Other research may be proposed if it is compatible with the mission of the 
Center and the topics suggested above. 

d) The final selection of specific research topics should be made to maximize 
the Center's overall impact in its mission area. 



Organization and Staffing 
Specifications: ShoaldsAiays 

1. No organizational arrangements are required beyond the general provisions of 
this announcement. 

2. The Center may organize a network of schools or districts, if such networking 
will advance its work. 

3. The Center should be organized such that the various parts contribute for each 
other to enhance the wholeness of its work as an institution. 

4. Staff should possess expertise in the theoretical and practical perspectives 
involved in its research, 

6, Such staff members may be employed on a rotating basis, but the overall 
staffing capacity in this respect must be permanent. 

7. The Center should have a permanent, substantially full-time director. 
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Di ssaLnatlon 

Specifications: Shoulds/WLlls 

1. The following guidance is oriented toward the specific mission of this Center 
and it should be read along with the guidance on dissemination contained -in the 
general provisions of this announcement. 

3. It Td.ll want to be creative in getting news about its work to popular magazines 
& newspapers that will be read by parents and the general public. 

6. The? Center will be intimately acquainted with the needs of these constituencies 
ana ixet with them frequently to learn about their needs as well as pass on the 
results of its research. 



Collaboration with Other Centers 
Specifications: Shoulds 

1. The Center should be alert to collaborative opportunities with other NIE 
Centers and plan to allocate resources to them in accordance with guidance 
contained in the general provisions of this grant information package. 

Eza^les 

3. Teaming (higher cognitive skills) 
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This document provides a summary analysis of the concept and issue papers 
developed for consideration by the R&D Center planning team. The papers are 
categorized by their relevance to four R&D Center program areas. Each paper 
is listed by title, followed by a brief statement of its purpose. The speci- 
fic issues or problems identified in the paper are noted and, in most cases, 
summarized. If the paper contains specific implications for proposed research 
bearing oti the identified problem, this also is noted and summarized. In some 
papers, no implications or proposals for research are offered. 

I. STAHDARDS 

A. Standardg for Educational and Psychological Testing - George Nadaus, 
Bog ton College 

This paper deals with the Standards for Educational and Psychological 
Tes ting published by the American Psychological Association and prepared by 
the Committee to Develop Standards for Educational and Psychological Testing 
of the American Educational Research Association, the American Psychological 
Association, and the National Council on Measurement in Education. The paper 
describes the content of each of the four major sections of the Standards and 
presents implications of each section for the work of the Center. 

ProbleBS or Issues 



The paper discusses each of the four major sections of the Standards. 
The content of each section is briefly described. It is noted that the 1985 
standards will impact educational testing for some time to come, both as a 
result of their use in court cases and through their use by those in educa- 
tional testing programs at the state and local levels. 

Implications or Proposed SolutloBta 

Part I, Technical Standards for Test Construction and Evaluation, pro- 
vides the basis for the Center's development of practical techniques and 
checklists for school districts thfi\t are building their own criterion- or 
curriculum-referenced tests. Practical checklists for use by local school 
personnel can be developed for a variety of uses at a variety of levels. 

Part II deals with the proper use of tests in different situations. The 
translations of the Standards into guidelines and checklists for a variety of 
audiences will help these different user groups appropriately interpret and 
use t3st scores^ The section dealing with test use in special education is 
especially pertinent to the proposed Center. Also, the chapter on program 
evaluation may lead to other practical checklists and materials detailing the 
better use of test results as part of an agency's effort at program evalua- 
tion. 



Part III, Standards for Particular Applications, notes that the two 
chapters in this section serve as the benchmark for the NIE Center. The two 
chapters are "Testing Linguistic Minorities" and "Testing People Who Have 
Handicapping Conditions." The Center should use this material to develop 
practical guidelines for local educators on how to modify and administer tests 
to students who have different handicapping conditions and who come from 
different linguistic communities. 
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Part IV is Standards for Administrative Procedures. Again, the NIE 
Center should use these chapters to develop practical guidelines for use by 
school officials regarding test administration, scoring, repoi.ting, and 
protecting the rights of test takers. This paper basically suggests that the 
Center use this set of standards as the basis for developing checklists for 
materials that translate the applicable standards for users and provide 
guidelines for their implementation. 

B. Setting Staadarda - John Poggio^ University of Kansas 

This paper addresses the general problem of standard setting at both the 
state and local levels. The paper summarizes a review of the literature that 
reveals that no fewer than 30 methods for setting standards on test perform- 
ance have surfaced and have been used within the recent past. 

ProbleMs or Issues 

^- Exploration of alternative standard-setting methodologies . Although 
there are many available methods, most are narrow in focus or 
limited in practicality. The paper suggests several problems that 
could be investigated in this area. 

2- Establishing theories for standard setting. The paper notes that 
there is little theory underlying the various methods and suggests 
that theory needs to be formed so that we can better understand the 
results of the various processes or methods. 

3- Implications of standard setting. A number of major and pressing 
problems that follow the setting of standards and deserve research 
attention are noted. 

Psychomet ric Properties, Finally, the paper notes that research is 
necessary to broaden our focus to a consideration of the methods or 
sets of methods used in setting standards. 

tL&viev of the Effects of Standardised Testing - Georg e Madaus. Boston 
College 



This is a position paper, the purpose of which is a critical review of 
the literature on the effects of educational testing at the elementary and 
secondar3r levels. It is noted that the review is primarily concerned with the 
effects of traditional, school district, standardized testing programs, 
although the impact of newer state testing programs is included when appro- 
priate. 



Problens or Issnes 

The paper details severfkl important issues related to the effects of 
standardized testing and highlights implications for the v/ork of the Center. 
These issues are: 

1- The difficulties in evaluating the literature on the con sequences of 
testing . "~ ^ 
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2. 



Perspectives on the criticisms of tests. 



Characteristics of tests and the context of testing. This section 
considers seven crucial dimensions along which standardized tests 
can be differentiated. 

The consequences of tes ting . The consequences of testing are 
discussed under the following categories: 

a. School-level effects 

b. Teacher-level effects 

c. Student- level effects 

d. Parent-level effects 
laplications or Propoaed Solutions 

The paper suggests one or more implications under each of the sections. 
The following summarizes suggested work for the Center. 

1- mfficultles in evaluating the llteratwre on the consequences of 
testing ; : 

a. Analyze different philosophical and ideological arguments 
underlying the debate about testing. 

b. Engage various stakeholder groups in dialogue to help under- 
stand the social and political dynamics of standardized 
testing. 

2m Criticisms of tests 

c. Gather empirical data that clarifies, counters, or substanti- 
ates the various criticisms of tests; explicate the competing 
belief systems associated with support and criticisms of tests; 
and work with LEAs to reduce reasons for criticisms, especially 
related to special populations. 

3. Characteristics of testa 

d. Work with LEAs and SEAs to help them distinguish between 
different objectives cf measurement and to evaluate how tests 
can be used across different objects of measurement for dif- 
ferent decisions. 

e. Work with school districts to develop content tests for subject 
matter areas. 

f. Investigate strengths and weaknesses of norm- and criterion- 
referenced tests relative to various uses with different 
audiences. 
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Help educational agencies distinguish among the various uses of 
test results and the various inferences they may draw from test 
scores. 



h. Investigate different reporting strategies to communicate test 
results. 

i. Investigate the impact of external testing prograiuS on teaching 
and learning. 

j. Study the levels of aggregation of test scores. 

k. Be continually alert to the interrelatedness and interaction 
among the dimensions specified in this section. 

4« The consequences of testing ; 

i- School-level effects 

(1) Conduct in-depth studies in school districts to ascertain 
the effects of various kinds of testing programs on 
decision making. This study should be conducted with 
various kinds of districts, utilizing different testing 
programs. 

(2) Design report forms and software packages that will 
provide administrators with useful analyses of test 
results . 

in. Teacher- level effects 

(1) Survey teachers from various states about their percep- 
tions of external tests and the effects of such testing 
programs on the instructional program. 



(2) 



Survey teachers working with different parts of testing 
programs to determine the effect of these programs on 
teacher perceptions of test relevance and use. 



(3) 

Student- level effects 



Study the effects of test information on teacher expecta- 
tions and teacher behavior. 



(1) Investigate the impact of providing students with test 
information and the effect of testing prograr.s on dropout 
ra te. 



(2) 



Examine how measurement-driven instructional programs 
affect student behavior. 



(3) Monitor the programs over time. 

(4) Document effects on students of providing them and/or 
their teachers with test information. 
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(5) Develop a research program that examines test effects with 
different types of student groups. 



o Parent-level effects. Develop a program of research aimed at 
parents of different kinds of backgrounds and their under- 
standing of different kinds of test information. 

D- Professio nal Standards and Equity; The Role of E valuators and 
Researchers - Carol Tittle 



This is a paper prepared for the 1984 AERA meeting. Three major sets of 
standards are examined in the paper for their relevance to equity concerns in 
education. The standards are: 1) The Joint Committee's Standards for the 
Evaluations of Edu cational Programs, Projects and Materials ; 2) The Evaluation 
Research Society Standards Committee's Standards for Program Evaluation ; and 
3) an edition of the Joint Technical Standards for Educational and Psychologi- 
cal Testing by the Joint Committee of the American Educational Research 
Association, the American Psychological Association, and the National Council 
on Measurement in Education. The author also proposes equity standards for 
evaluators. The paper initially examines some contextual influences on 
professional standards and then reviews the issue of equity. This section 
contains many examples of how standards attempt to address equity issues. The 
author concludes that the test standards are responsive to equity concerns. 
She presents a thorough discussion of the two sets of evaiuaticn standards 
with regard to their concerns over equity. The paper concludes with a pro- 
posal for a set of equity standards for evaluators. 



II. PSYCfe^ETRIC THEORY 

A- Patterns of Iteui Reaponse - n. David Miller, Pnlverslty of Kansas 

This paper discusses recent research on patterns of item response and the 
kind of useful information about student test performance they can yield. The 
paper reviews research on item response patterns and particular indices that 
have been developed to measure patterns of item responses made by a student. 
The paper focuses especially on uses of these indices of item response pat- 
terns as a diagnostic tool. The remainder of the paper summarizes recent 
literature related to the indices of patterns of item responses. 



^- Crlterlon-Rgferenced Testing; Research Directions - Dou glas Glasnapp, 
Dnlverslty of Kansas ~ " — 

This paper contrasts nor.m-referenced and criterion-referenced tests. The 
paper notes that the criterion- referenced/norm-referenced distinction has 
become, at best, clouded. Discussion presents the advantages and distinctions 
of each type of test. The literature is reviewed regarding several issues 
involved in the norm-referenced/criterion- referenced distinction. 

Probleias or Issaes 

The point is made that there is a wealth of background information, 
supported by empirical data, on standardized norm-referenced measures. 
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Although the theory of criterion-referenced measurements indicates what 
characteristics test scores should have, the limited history has not provided 
us with comparable empirical information. Questions need to be raised and 
answers provided regarding what test score characteristics can be anticipated 
from criterion-referenced tests in the context of broader applications. Do 
they function as expected, based on the theoretical underpinnings of the test 
construction procedure? 



LBpllcatlons or Propoaed Solutions 



To answer the questions raised in this paper, research needs to be 
directed to specific technical issues on various aspects of criterion test 
construction. Research must be focused on the application of existing 
criterion-referenced methods. The application orientation for criterion- 
referenced testing research may be viewed as a focus on validity investiga- 
tions into criterion-referenced test development procedures and resulting test 
score outcome characteristics. Therefore, research should address all phases 
of a testing program, including test purposes, levels of use, test development 
procedures, reporting formats, score interpretations, and intended audiences. 



^- Test Deglgn; Cognitive Models of I tea Res ponse - Susan Embretson 
Balverslty of Kansas 

The paper presents a specific proposal dealing with the design of educa- 
tional tests. 

Probleas or Issues 

The goal of the project presented in this paper is to improve the design 
of educational tests by explicating the cognitive processes that determine the 
individual's response to test items. The goal of the proposed project is 
further described as providing a foundation for the design of specific educa- 
tional tests by assessing the cognitive components involved in important item 
types. A secondary goal is to study procedures in psychometric models for 
dynamic testing. The paper discusses the traditional practices of test design 
and item specifications. Two disadvantages of current practices that bear on 
the proposed project are presented and discussed. 

iBpllcatlons or Propoaed Solationa 

The proposed project combines methods of cognitive component analysis 
with latent trait models that calibrate the impact of cognitive components on 
item responses. The recent background research supporting this approach to 
studying item response is presented in some detail. 
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III. USE OF OF TESTS 



A. The Natural Use of Student Te8tJ.ng and Evaluation in Schools by Classrooa 
Teachers and Principals, and How That Use May Be Enhanced to Inprove Tea ching 
and Learning at the Local Level - Jaaes Sanders, Western Michigan University 

This paper offers a proposal for studying the naturalistic uses of tests 
and evaluations in schools to develop prototype procedures and products for 
enhancing such use. 

Problems and Issues 

TJie paper concludes that neither testing nor evaluation has been inte- 
grated into the everyday practices of classroom teachers and administrators. 
It presents several responses to this situation and describes some naturalis- 
tic inquiry that has been pursued relative to this situation. The basic 
question is how testing and evaluation practices can be tailored to fit the 
natural information needs of school teachers and administrators to help them 
make better decisions and to improve the quality of instruction. 

laplicatlons or Proposed Solutions 

Three procedures are offered tj study the basic questions presented. 

1. Case studies will be conducted in school districts of varying sizes. 

2. Based on the information available from the case studies and from 
the literature, prototype procedures and products will be developed, 
aimed at enhancing testing and evaluation routines in classrooms. 
Each procedure or product will have student outcomes associated w< th 
it. 

3. The prototype products or procedures will be prepared in final form 
for large-scale dissemination. 



^- lasues in Testing in Bilingual Education - Robert Mendr o, Dallas Indepen- 
dent School District ' " • 

This paper addresses the problem of non- English-speaking students enter- 
ing the public schools in the United States and focuses on the use of testing 
in bilingual education. Student testing is a major component of programs la 
bilingual education. It is used in student selection, the evaluation of the 
success of students, and as a measure of effectiveness of the curriculum. 

Probleas or Issaea 

Problems in testing students in bilingual education programs arise from 
two primary areas--the availability of tests and the adequacy of these tests 
for their intended purposes. There is a lack of tests in most other languages 
other than Spanish. Publishers give very little attention to this problem 
since the number of tests needed is so small. The problem of test adequacy, 
however, is the more serious problem in bilingual education. Critical con- 
cerns in this area include determining the functional level of a student's 
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English ability, measuring student performance against a standard in a 
lang-iage other than English, and adequately testing a student at his or her 
particular stage of development. 

laplicatloas or Proposed Solutloas 

Research efforts are needed in the process of learning English as a 
second language and related testing issues. Both research efforts and test 
development programs will be required to solve problems in this area and to 
move measurement in bilingual education to a more precise and meaningful 
level. 



C. Use of Testa la Policy - George Madaus, Boston College 

During the past five years, in a period of reform in education, tests and 
test results have been increasingly employed in a variety of ways, particular- 
ly at the state level. The use of tests in the policy sphere is a growing 
trend, according to the information presented in this paper. The paper high- 
lights issues inherent in the policy use of tests and develops implications 
for the work of the NIE Center. 

Probleas or lasiies 

There are two principal uses of tests in the policy sphere. The first is 
the use of test information to inform policy makers, and the second is the use 
of tests as administrative devices in the implementation of policies. Both 
issues are addressed in this paper. In the case of informing policy makers, 
test results are used primarily to describe the present state of education or 
some aspect of it, or in lobbying efforts. Some of the recent educational 
_^form reports have used test results to bring attention to what they consider 
the mediocre state of American education. Also, in state-wide testing or 
assessment programs, results are published and comparisons, whether intended 
or not, are made. Although decisions may be based on these tests, the paper 
notes that little is known about how these assessment results affect the 
curriculum, teaching, or learning. We do know that test results have fueled a 
debate on the need for educational reform and that some school districts offer 
programs to better prepare students to take tests. 

In sharp contrast to the use of tests to inform, the administrative use 
of test results triggers a direct action, on either an individual or an 
institutional level. In many states, test results directly drive a variety of 
programs ot actions. When tests are used in this way, the paper notes that 
their Impact on teaching and learning at the local level is a direct function 
of the nature, magnitude, and immediacy of the rewards or sanctions involved 
Eight positive effects attributed to the use of external tests as administra- 
tive mechanisms are presented, followed by eight negative effects. Since 
there are both positive and negative aspects to this use of tests, monitoring 
procedures need to be developed. 

Ijipllcatlons or Proposed Solutloas 

The paper presents implications for the Center in both the areas of tests 
used to inform and tests used as administrative devices. 
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!• Use of tests results to inform policy 



a. Develop monitoring procedures. 

b. Study how test results indirectly impact on education at the 
state and local level. 

c. Identify variations in this use and study in more depth at 
selected sites . 

d . Based on this work, prepare materials and techniques that 
various groups can employ to help them interpret test data and 
assessment results. 

e. Use the results to plan curriculum and instructional changes. 
Use of tests as administrative devices in p olicy 

f. Use the 16 positive and negative effects e - j to external 
testing programs as working hypotheses t Investigations 
of impact at the local level. 



g. Use both survey, research, and in-depth case studies to study 
the effect of programs across different types of school sys- 
tems . 



h. Pay particular attention to positive and negative impacts on 
minorities, bilingual, handicapped, and learning-disabled 
students . 

i. Identify and analyze uses of tests as administrative m< lanisms 
both in this country and abroad. 

j. Develop practical materials and techniques that LEAs can use to 
evaluate the local effects of testing programs and improve 
their instructional delivery systems. 

k. Develop techniques and materials for SEAs to use to evaluate 
the psychometric characteristics of their tests. 

Factors Affecting the Utility of Standardlxed Tests - Ma ria Lalsa 
Goazaiez, Dallas Independent School District 

This paper addresses a number of factors affecting the use and value of 
standardized testing programs in the public schools. The paper discusses 
purposes for testing and factors involved in proper test selection. 

Problems and Issues 



D« 
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The paper points out five major factors to be considered in assessing the 
utility of tests. These are: (l) psychometric considerations, (2) administra- 
tive considerations, (3) services provided by the publishing company, (4) 
scoring and reporting considerations, and (5) testing costs. A number of 
issues underly each factor and emerge as important considerations in the 
utility of tests, leading to a variety of potential research problems. 
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lapllcatlons or Proposed Solutions 



In order to facilitate the use of test results and to assure sound test 
selection practices, a number of potential research studi-^ need to be under- 
taken. The following studies are suggested in this paper: 

1. A study is needed on the degree of instructional validity and diag- 
nostic utility of norm-referenced testing. 

2. Research is needed to link testing with curriculum and instruction 
in a practical manner. This might involve using norm-referenced 
data to produce objective, reference- type data or enhancing national 
norm-referenced data by the addition of more specific information 
related to district objectives. 

3. Research should be undertaken to help uncover the degree to which 
different commercially marketed test-taking materials actually 
prepare students for testing. Within this framework, the quality 
and use of commercial testing instruments need to be assessed. 

4. An examination should be considered on how test items and responses 
can be made more secure nationally. 

5. New avenues of testing, reporting, and scoring need to be identi- 
fied . 

6. Procedures that have been established in different districts should 
be studied to further delineate successful test program character- 
istics. 

^- Helping Claaarooa Teachers Use Testa and Testing Result s - Cordelia 
Alexander, Dallas Independent School District " 

This paper posits a renewed commitment to testing and, as a major aspect 
of this renewed commitment, recognition of the classroom as the focus of 
testing and test use. Teachers play a central role at this level, through 
administering tests, receiving the results of student performance, and using 
the results to guide instruction. 

Probleaa or Isauea 

If testing affects students, presumably the effects often occur because 
teachers use test results in some consistent way. This paper addresses three 
particular issues that must be considered in helping classroom teachers 
utilize test data more effectively: 

1. Issues in the various uses of this data 

2. Procedures for helping teachers use test information 

3. Other areas that need research 

A number of issues and particular problems are addressed in each of these 
three areas. There is, for example, an increased responsibility to communi- 
cate the limitations of tests, to consider their effects on instruction, and 
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to influence proper test use. Also, teachers rely heavily on tests they 
themselves have developed and on their own observations of student ^rk. 
Similar problems exist in helping teachers use tests properly. The paper 
notes seven major tasks that should be included in an assistance program for 
helping teachers use tests. 

iMplicatioas or Proposed Solutions 

In developing a series of research questions, the author of the paper 
offers questions presented by two other writers, Rudman and Rudner, who have 
summarized the following recent concerns on test use in the classroom: 

What are alternatives to testing for students diagnosis and assess- 
ment? 

How can one create a climate that fosters better test use? 

How much do p^^rents know about testing? What are their attitudes? 
What are the best means of communicating student achievement? 

What are student attitudes and reactions to testing? 

Do teacher-developed tests cover a r/inge of cognitive skill levels? 
Are they valid, reliable, and effective? What can be done to 
improve teacher-developed tests? 

Do curriculum-embedded tests cover a sufficient range of cognitive 
skills? Are they valid, reliable, and effective? What can be done 
to Improve them? 

What are characteristics of useful diagnostic tests? 

How can tests be used to provide better diagnostic information about 
students, groups of students, and curricula? 

How can teachers develop better diagnostic instruments? How can 
test publishers develop better diagnostic instruments? 

What are alternative ways of reporting test results to students? 
Parents? The press? 

How can local school districts better analyze test results? 

How do teachers use standardized tests diagnostically? 

What are the appropriate uses for each of the different kinds of 
tests? 

How can the reporting of test results be improved? 

What are some successful patterns of in-service education that could 
be replicated for training teachers in the use of assessment data in 
the classroom? 



^* Position Paper on Assessaent for Special Education Studen ts - ThoMS j. 
Helry, Dgllas Independent School District — — 

This paper focuses on the need for assessment instruments for mildly 
handicapped persons. 

Problems or Issues 

The paper, citing other sources, makes a distinction between traditional 
assessment, used for classification and eligibility determination, and con- 
temporary assessment, used for individualized program planning and implementa- 
tion. The implications of this distinction are especially important in the 
assessment of the mildly handicapped. Many issues associated with the assess- 
ment of the mildly handicapped have not been resolved and are detailed in this 
paper. The mildly handicapped category includes the learning disabled, the 
educable mentally retarded, and the slow learner. Many problems exist with 
the assessment of these students, since they are often assessed in conjunction 
with the moro severely handicapped. The utility of traditional psychoeduca- 
tional assessment has been questioned on grounds that (a) etiology cannot be 
reliably determined using traditional measures, (b) etiology has been shown to 
be linked with a type of educational intervention, and (c) traditional assess- 
ment for special education eligibility is costly. 

An assessment of tha mildly handicapped is best viewed as a two-function 
process. Traditional psychoeducational assessment is utf.lized in conjuntion 
with other data about the students to make critical decisions concerning the 
presence or absence of a handicapping condition. Contemporary curriculum- 
based assessment is utilized to make individualized, educational, and pro- 
gramming decisions. 

Iwpllcatlona or Propoaed Solutlono 

The research questions are viewed in relation to an understanding of 
special education assessment as a two-function process. The research questions 
address diagnosis and how diagnosis leads to better contemporary assessment. 

1. The major task for research in the field of special education 
assessment is the opera tionalization of what constitutes a reliable 
and valid diagnosis for learning disabilities. 

A. What are the variables that discriminate a learning-disabled 
student from other problem, or handicapped, learners? 

B. What is a clinically useful approach to determine a discrepancy 

between ability and achievement? 

2. The design and psychometric properties of curriculum-based tests in 
this area is another area for research. 

3. "Exit criteria" for dismissing a student from special education after 

instructional efforts have been implemented is another area for 
future research. Correlates of successful mainstreaming should also 
be examined. 
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®' I«pllcatlon8 of Sp ecial Needs Populations and Aaat ssm ent: Practices - 
Edward Meyen, University of Kansas "~ ~ 

This is another paper addressing issues related to problems of handi- 
capped children and youth. The focus of concern is the appropriateness of 
assessmen'. instruments used, and the determination of eligibility for place- 
ment, in special education programs. The paper clearly and cogently discusses 
the special needs population and the academic context. This discussion 
provides a frame of reference for exploring assessment-related research issues 
as they pertain to the special needs student population. 

Ijpllcationa or Proposed Solutions 

The paper provides the following list of statements that can be translat- 
ed into research questions to provide the basis for systematic investigations 
of how best to assess the academic performance of special needs students. 

1. Children with disabilities tend to be subjected to more testing than 
other students, but, except for the mildly handicapped, they are 
rarely tested in basic academic skills. 

2. Because of the variability within school districts in policies 
governing the administration of academic skill assessment tests as 
they apply to students with disabilities, aggregate results of 
testing programs at the district level may not be representative of 
students enrolled. 

3. Teachers of mildly handicapped students are oriented toward teaching 
for mastery and thus tend to favor a criterion- referenced testing 
approach. 

4. The assumption is often made that oral administration of tests to 
this group compensates for the limitations they may have in reading. 
Results of research in this area are mixed. 

5. For the mildly mentally retarded, and to a certain extent for 
students who are emotionally disturbed or learning disabled, lack of 
motivation is considered to be a major contributor to poor academic 
performance. A student's history of failure In learning and testing 
situations causes the student to accept failure and not to compre- 
hend the significance of assessment exercises. 

6. Teachers often exercise care in the amount of work assigned to a 
student for completing in one sitting and may also provide prompting 
in this work in the instructional program. A modular approach to 
the presentation of test items tnd allowance for some level of 
prompting merit exploration as methods of building confidence in 
testing Situations. 

7. There is a tendency with handicapped students to introduce the 
application of academic skills into the curriculum as early as 
possible. The concern for application is commendable, but it may be 
occurring at the expense of teaching the necessary basic skills and 
may limit the scope of concepts taught. 
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8. The language structure used by children with intellectual impair- 
ments is less developed than in non-disabled peers. in developing 
test items, reading difficulty often requires primary attention! 
Simplification of the language structure when assessing more complex 
skills may be effective with this population. 

9. Computerized testing procedures that combine testing with remedia- 
tion should be explored. " 

LeglaUtive lapa ct and Educational Refora - Florida PDK Conaortlum 

This paper is a series of related documents that resulted from the work 
of the Florida PDK Consortium Planning Task Force. it reviews the history of 
programs in Florida, among which are various testing programs for students at 
all levels. it summarizes the concern of a numbc-r of education and societal 
groups regarding the programs and references, criticisms and issues, that have 
appeared in the public media. It presents a plan for collaborative study of 

pforfl°^"?K ^^'^ °^ P""" ''^■'■''^ ""'^^ programs can be implemented in 

Florida. The paper should be read ia its entirety for full understanding. 

The State of Florida has been at the forefront of legislating educational 
reforms during the past decade. Beginning with the Educational Reform Act of 
1976 and continuing through the Management Training Act of 1983, legislative 
decisions have focu ed on the setting of student standards, teacher effective- 
ness, administrator effectiveness, and curriculum and instruction. 

As Daniel Duke has noted in his recent essay on educational excellence. 

Hard on the heels of most reform movements in education come 
the demands to know whether or not the reforms have "made a 
difference" (Duke, 1985, p. 671). 

Accountability for a major investment of resources into legislated reforms is 
certainly one reason for undertaking an evaluation of recently legislated 
educational reforms in Florida. An even more compelling rationale for this 

fllZ: ^'^T^'V' T ""'^ P"*^ experience so that current and 

tuture efforts at reform may proceed from an expanded knowledge base. 

There are reform packages already near completion in several state 
*V planned for the coming year (Pipho, 1985). Current 

iTfho ^ K %^ V^""^ ^"^""^ activity in bellwether states is proceeding 
without benefit of systematic and thorough understanding of the consequence! 
«frr?n \ teaching and learning. Since student testing and standard 

setting have been used as instruments to bring about desired changes in 
Florida, an evaluation of these efforts will have significance for the policy- 
shaping communities of every state considering the use of testing and standard 
setting to reform its educational system. 

Thus, the research problem to be addressed is, "How have legislative 
testing and standard- set ting programs in Florida affected significant stake- 
holders anrt critical institutional functions in the education System?" 

The purpose of a project would be to collect empirical information about 
the impact of student testing and standard-setting mandates or acts legislated 

121 



in one state , Florida, that has been active in education reform, and the 
impact of its testing programs on important stakeholders in the state and on 
institutional functions within the state education's system. 

Specifically, this project could result in: 

1. descriptions of documented effects found in selected schools and 
school districts 

2. descriptions of changes in student needs and achievement over a 
ten-year period 

3. recorded testimony and analysis of this testimony for individuals 
and groups affected by the legislation or mandates; 

4. analysis of reports collected within Florida and from other states 
- in which the impact of mandated or legislated testing and standard- 
setting programs have been studied 

5. survey of findings of reported impacts on stakeholders and institu- 
tional functions throughout the state; 

6. interpretations and recommendations by an expert panel on the use of 
testing and standard-setting legislation and state mandates to 
improve the education of students 

7. a model for state-wide evaluation of educational reforms through 
mandated or legislated testing and standard setting 

Such a project could provide answers to the following questions: 

1. What documented impact was found for testing and standard setting 
legislative acts and state mandates on students, parf^.nts, adminis- 
trators, communities, and business/industry? 

2. What documented impact was found for testing and standard- set ting 
legislative acts and state mandates on educational policy making and 
planning, finances and facilities, curriculum and instruction, and 
school /community relatins? 

3. What aspects of state mandates and legislation appear to have the 
greatest impact on stakeholder groups and institutional functions? 

4. What research-based recommendations can be made to states regarding 
current and future legislation and mandatesi that involve student 
testing and standard setting? 

5. How can states best evaluate the impacts of l<!!gisiation and mandates 
that involve student testing and standard setting? 
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^* Inflaence of Tr&inlng in Meaaureaent Skills la Higher Ed ucation by Mary 
App.e Banda Western Michigan Dnlversity ~~ 

The paper presents the background and structure for a research project 
dealing with the influence in training measurement on the uses of test in 
school. Background is presentee for the need to improve uses of test based on 
current research, writing, and state activities. The paper presents four 
proposition related to testing and assessment in schools. Two of the pro- 
position are relevant generalizable tc post secondary institution. It further 
points out that higher education in general is not prepared to deal with 
training the area of evaluation and assessment. Evaluation work needs to be 
undertaken both the current status and the use of tests in higher education. 

Inplicatlons For Projects 

There are four objectives for the research proposed in this paper, (l) a 
preliminary identification of patterns of test constructs and use will be 
investigated in higher education. (2) the identification of barriers to 
training in testing in higher education will be attempted. (3) an analysis 
of programs designed to promote testing m higher education will be completed 
(4) Models for training in higher education based upon the finding from this 
project will be developed. 

The project is conceived as a one year project using naturalistic in- 
quir techniques. The study will result in reports on research activities with 
can be disseminated through the center, structured instrumentation which can 
be used to study the same phenomena in a larger sample of institutions, and 
training material that can be used by center personal or disseminated for 
broader use in higher education facilities. 

J. Judic ial and Legislative Inflaences in Educational Testing ~ Plane Pullin 

This paper is concerned with the influence courts and legislatures have 
upon educational policy makers, especially in light of the increased uses of 
tests for classifying students. The paper first discusses the involvement of 
the courts and legislatures in educational policy and describes relevant 
cases. Courts have begun recently to scrutinize the tests themselves and this 
development also is reviewed. Most recent court cases have resulted from 
testing requirements that have been legislatively initiated. Two primary 
areas addressed by legislation have been tests used to determine program or 
individual educational accountability and tests used to determine eligibility 
for special education services. Cases and issues relevant to this latter are 
described in some detail. The paper then identifies nine legal issues emerg- 
ing from legislative and judicial decisions. 

^- gqalty la soea and Stuident Testing - Arnold Galle goa. Vestexn nichlgan 
Dnlversity ' ' — = 

This paper addresses one important problem associated with increased use 
of testing to assess student progress and to classify students as part of the 
effort to correct problems with instructional programs in the schools. 
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Problems or Issues 



The primary focus K}i this paper is on a seiTi^a of equity questions, ihe 
first issus described ift the paper is curricUlar equity--the balance in 
teaching the basic skilj-s which are tested and cHCting back on other areas 
study. The issue of wjlat subject areas received top priority also leads to 
problems of equity in professional roles. The t^i^d area discussed is that of 
social and cultural equit/ and the impact of testing on this issue. 

Implications or Pr^ jpcy Beci Solutions 

Two research issued of relevance to the pi;o^>osed center are suggest^^' 
(a) work on test, construction and response tfodes; and (b) assessing t^\ie 
impact tests have on the curriculum and teachers, 

IV. THE DEVELOPMENT OF ^^W STSTEMS AND TECHNOLO(?X^ 

A- Comptnter Aaslgted P jrofejisloiml; A Propog» l to Develop an Informati W 
System to Asalst Schoo F^Profe aalonals to Plaij gd&g an d Implementlgg S cfa^Ot 
I«prove«ent - William Co ^Ij sy, Pniverstty of Pitt^jg^rgh " " ^ 

During the 1984-85 School year, members the Learning Research ^hd 
Development Center hav^ been working with tha Pittsburgh Schools in the 
development of a protot^ype microcomputer syste^ti In one elementary school. 
This paper presents an analysis of the need fot an automated information 
system in a local school. It traces the use of automated information syste^^^ 
from the 1950s and 1960^, i. d discusses applica tl^oUs for centralized, syste^^- 
wide functions. It is ptoposed that, with th^ arrival of inexpensive h^t 
powerful microcomputers, automated information sy^^tems can be used effectively 
in local schools. Th^ general goals of such e^^tema are (l) to improve 
student achievement, (Z) to enhance the quality of school life, and (3) to 
provide equal learning (opportunity for all stude^its. The functions of t^e 
automated information sy^t^m at the local school Wilding are also discu3sea^ 

Probleaa or laauea 

To make an information system useful at the local level, its characteris- 
tics need to be designed specifically for local u^e- Issues of dependability^, 
accuracy, and currency of Information files are discussed. The integration 
several information functions at the local level 1^ presented as an import^At 
outcome of the developm^at>^l work. With these p^^obUms in mind, staff of the 
LRDC designed and implemented a prototype syst^W in one elementary schoAi 
during the past year. the feasibility of sucp a school-based information 
system for improving infarmatlon flow and informaUou use has been demons trat^- 
ed; however, considerable work needs to be done if such systems are to operate 
without the day-to-day p1^eSence of a technical support team. Also, the job Of 
building into the systei^i the kind of expertise that will assist the prof^^- 
sionals in each school al^ill remains. 

lapllcatlona or Pr ogo gi^d Solutioaa 

The proposal preset^t^d in this paper is to futther develop and test th^s 
prototype system in the context of the Pittsburgh Schools. Two major taaKs 
are proposed: 
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1. to (f^^efully test the notion that the system currently in place can 
be operated successfully by personnel in the school with no 
addi-cional staff requirements; to place the system entirely in the 
hand^ of the school personnel and to study what happens 

2. to «^6t)H(;ate the initial implementation in another school in the 
Pitt;^l?urgh Elementary School System 

Compate»r.v.ftftU erated, Personalized Testing - George H. Olson. Dallas 
Independent Sg V.ol Platrlct 

This papAf briefly presents the possibilities of developing and using 
school system c<imputer data bases for student evaluation. The writer notes 
that the ideas ptese&ted can only be developed in a context of a large-scale 
research and development effort. 

ProbleMs fyt^ Is jtues 

In large school districts, the current use of centralized information 
systems and th^ linKlng of student evaluation to test scores on nationally and 
locally develo(i«<l standardized tests presents special problems in the adequate 
evaluation of ^tudetits. Student evaluation generally is reduced to an annual 
set of observe t^ions. Further, the validity of these observations with respect 
to individual ^Cudeflts depends to a great extent on the correspondence among 
the district's Curriculum and instruction objectives, the student's level of 
ability, and the skill and ability domains measured by the test. This often 
results in a Usa- than-adequa te assessment of individual students and a 
needless waste o( valuable instructional time. 

Impllcatl ^tiA o g Proposed Solatlons 

The papet proposes an efficient system for generating unique, personal- 
ized tests tailored to individual student characteristics, but which still 
retain the properties of standardized tests allowing for normative or compara- 
tive evaluatioi^. Jn the system proposed, the construction and printing of 
individually u«lique tests would be accomplished on the mainframe computer. 
The system would be based oc recent technological advances in test-related 
fields such ^ft item response theory, adaptive testing, computerized item 
banking, matrl.;^ sampling, etc. The paper presents a brief overview of the 
system and dis<JU«se3 the advantages of the approach. 



^' The Progr^x on Technology m Student Testing, Evaluation, and St andards - 
Richard Frisbi<? 4nd_ I.yke Thoapson, Western Michigan Dniyerai^ ' 

This pape!^ presents opportunities that currently exist for applying 
technological ^pt>Ucations to testing, evaluation, and standards in local 
settings. 
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Ijaplicaeiona or Proposed Solatlcms 



Three potential projects are suggested in this paper: 

1- Research and development on school district data bases . The specif- 
ics of this project are presented elsewhere in the institutional 
proposa 1. 

2- The expert advisor series. This proposal suggests an "expert 
systems" approach for developing technical assistance materials 
aiding consumers at the local level. The expert adviser series of 
materials would be a growing collection of expert system computer 
programs dedicated to helping producers and consumers at the local 
level better understand, use, develop, and improve testing, evalua- 
tion, and standards for student academic performance. The activi- 
ties suggested for this project would include (a) the identification 
of areas of expertise in testing, evaluation, and standards that can 
be provided to local users and that can also be implemented on a 
microcomputer-based expert system, (b) the selection of at least two 
areas for prototype development, (c) the development of the expert 
adviser in these two areas, (d) field tests of the expert advisor 
materials and subsequent revision, (e) dissemination, and (f) 
development of new titles for the series. Two examples are provided 
in detail for the reader. 

3- _A programmers" library for computer-based testing and evaluation. 
This project consists of the development of a specialized micro- 
computer program library on testing and evaluation for use by 
consumers in the field. The project would focus on the necessary 
criteria and judgments for the selection of the materials. Stan- 
dards for use in this project would be developed. 



126 



19 



EKLC 



